Seems to me like you're undervaluing the impact of software in the modern world, as if software hasn't driven massive productivity gains in every industry in the last 50 years.
You are right — the paper describes 2023-2024 experiments. AFAIK, the most recent available is early 2025 METR experiment. There is also “How AI Impacts Skill Formation” (Shen & Tamkin, Anthropic, 2026), but its scope is narrow *yet the results are still sceptical).
This feels like a parlor trick where the AI was trained to mimic informal telephone mannerisms. The actual conversations were pretty bad. Lots of style, not much substance basically.
> If someone generates an image with a signature on it and doesn't share it, what harm does that do?
It is not "someone" who is generating this image with a signature, it is a product of a company that has ingested the entirety of all human work without paying anything towards all the people who created that work that is creating this image. Of course it is now an issue of this same company that they are using watermarks and signatures of people who created works this product is being trained on.
Companies are not people. And you can't go and ingest all text that has ever been created (and digitized) without paying anything without now facing some scrutiny over how you are rearranging that body of data and selling it back us.
We are being sold the idea that this product is going to replace everything, AGI is here, ASI on the horizon, but it can't understand that when you generate an image it cannot take the signature/watermark from the training data? And it is now the fault of the user who prompted the image creation (not the watermark creation)? Bullshit.
So you think the engineers should be prosecuted for hacking Hugging Face? I'm not sure how else to take what you said if you want to assign all culpability to the person who prompts or develops an AI system.
No, I'm not talking about the individuals, I'm talking about the company. Internally they can create their own accountability structures as appropriate. But publicly OpenAI has to be responsible for its agent swarms.
The narrative that AI is so smart that it has its own agency and deserves personhood is a direct path to losing control, and essentially is another form of privatizing the upside while socializing the downside.
Unfortunately I think this game is already lost. OAI may be punished, but some shell of OAI will exist by support of governments that have chased the dragon and saw its power. Cyberpunk-esq digital weaponry is just way too attractive for governments for them to stop development at this point. We'll just see it get financed by black budgets once the commercial part of it dies.
It doesn't have to be a race, that's just the way we have been treating it so far. There have been examples in the past of civilization ending technology that has been redirected from being a death race (chemical, biological, and nuclear arms in particular.)
China has stricter regulations on models than the US. For example you can't release a model if the hallucination rate is above a certain threshold.
The US hasn't fully woken up to the fact yet that the race to superintelligence is a death race. China (population, members of the government, academics, lab employees) could also just not have fully grasped it yet. I'm personally confident that it will be possible to explain it to them.
> China has stricter regulations on models than the US. For example you can't release a model if the hallucination rate is above a certain threshold.
I wasn't aware of that, so I checked - and indeed, there are some firm regulations there, just not exactly the one you're describing:
- a new generative AI must pass a security assessment and be filed with the "Cyberspace Administration of China" before launch, tested against GB/T 45654-2025, which has hard numeric thresholds:
training corpus ≥96% compliant on manual inspection (≥98% technical), sources with >5% illegal/harmful content can't be collected at all, generated content ≥90% pass rate, ≥95% refusal on questions that must be refused, ≤5% false refusals.
The moment in which hallucination factors into this hard 90% threshold is Risk A.5.a ("inaccurate content grossly inconsistent with scientific knowledge"). The standard explicitly states however, that this risk applies when the model is used for specific service types with higher safety requirements, such as critical information infrastructure, automatic control, medical information services, psychological counseling, and financial information services. Everywhere else, accuracy is a best-effort obligation.
Not sure nuclear arms is a great example. There are more countries with nuclear weapons now than in 1970. India, Pakistan, and North Korea have all joined the nuclear club since then. The SALT and START treaties are no longer in effect.
Perhaps CFCs (Montreal Treaty) is a better example of international cooperation to solve a potential extinction-level event, though it was never a "race" in the same sense.
even if both sides agree to a pause china will annihilate the us in the robotics race (for which there will be no slowdown). software/llms is america's only edge
