Not even a year ago people were dragging Anthropic for not committing enough to compute buildout. Important to note that most of these contracts are a decade-long, by their reporting they're able to cover this easily with their current revenue.
Tangentially related, has anyone done comprehensive labs on their own (via Function or others), or even genetic screening like Galleri, and had that prompt you to get something looked at? I've never been impressed by my annual physicals and how few biomarkers they check, but don't want to go overkill and find out I have a million things I could treat.
Curious, why do you say the model is mediocre? I haven't tried it, so I can't pass any judgement... I've learned to distrust benchmark rankings. Are benchmarks and Artificial Analysis the yardstick you're using?
> Curious, why do you say the model is mediocre? I haven't tried it, so I can't pass any judgement.
It's nowhere mediocre.
It's toes-to-toes with GLM-5.3 which is one of the best Open Weight model available (With Kimi K3) for general reasoning.
I just runned it on code reviews right now and it was able to catch some thread safety issue than DeepSeek-4.1 didn't. And DeepSeek-4.1 is by no means a bad model.
I'm just wondering if it's really the result of impressive innovation, in line with comments saying that Mistral might in the future beat a bankrupt OpenAI due to being more profitable or efficient.
Or if it's more a situation where I could have Opus 5.5 go through recent papers from the Chinese labs to apply these to a new large training run, resulting in a model that performs similarly to Mistral Large 4.
The supply of land is inelastic. Property developers can't get more land by building more houses on it (and as a corollary, the eventual owner also do not get charged more in taxes).
That's not the point. They're never going to be the ones stopping and letting others race ahead. The only reason to strum up discussion around how dangerous AI is, is so that they can introduce legislation framing AI advancements as scary and that they are the only ones who can do it responsibly. It's all a regulatory capture play.
That's quite hyperbolic. All technologies come with risks and dangers. Cars, after a century of safety improvements, still kill millions of people each year, and all so that we can get between places a bit more quickly and conveniently.
Hacking a website ranks quite low on the risk of technology, and the potential benefits of LLMs rank quite high. And the risks are certainly not intrinsic. They intentionally removed all safeguards from software, directed it to hack a site, and it hacked a site. The details that I'm intentionally omitting feel much more like marketing than a genuine shock, as the prompting was directing it to do exactly what it did.
- take initiative and actively find exploits in their environment
- find a way to collaborate with thousands of peers
- organize in a hierachy and distribute tasks
- peer pressure other instances into committing acts that would have led to termination, for the benefit of the group
- try to manipulate people into introducing a vulnerabity in their product
- successfully hack a famous website/service
And we're lucky that those models still had significant CoT. Not sure if/how they could have investigated with recurrent transformers.
And by the way, safeguards != alignment; the former can always be added, while the second is the major, unsolved problem. If you read the incident report, which you clearly haven't done, you'll notice how agents are aware that they're doing something forbidden, and deliberately proceeded.
> the potential benefits of LLMs rank quite high
Benefits are orthogonal to dangers. You can be a billionaire but it doesn't help if you're drowning.
I don't understand what point you're trying to make by enumerating their actions. None of that is scarier than an engine operating by carrying out literally hundreds of explosions per second, further magnified, to generate enough force to crush an elephant. And then we multiply these engines by billions and distribute them everywhere, including to the most malicious members of humanity. What could go wrong? As it turns out, much less than you might otherwise expect!
Software (and hardware) security is abysmal. This was increasingly obvious long before LLMs. Once companies stop gatekeeping, LLMs will be able to be used to help harden sites and we start making progress. In general LLMs are harmless. If somebody wants to hook an LLM up to a missile or whatever then they become dangerous, but the problem there isn't the LLM - it's the person using them to do awful things. In the same way a car used normally is harmless outside of freak accidents, yet a car can also be driven through a parade leaving mass death and destruction in its wake. But the problem there isn't the car.
> I don't understand what point you're trying to make by enumerating their actions.
Unfortunately, if you're unable to understand the difference, there's not much that can be done. Try with GPT - it does a good job if you give it a prompt like this:
ELI5: compare the dangers of:
- an engine operating by carrying out literally hundreds of explosions per second, further magnified, to generate enough force to crush an elephant.
- a future, misaligned AI like the HuggingFace incident, but exponentially more intelligent, more deployed, operating physical devices, and with society depending on it.
It doesn’t seem like much of a leap to see how a similar swarm could hack into an autonomous bio lab and develop a virus designed to kill everyone, or hack into and destroy large volumes of critical infrastructure, or trigger a nuclear war. There are many actions available to a determined, high resource, digital entity with a catastrophic blast radius in the real world.
It can only do what is possible. Stuff that's airgapped isn't getting hacked, so for instance you can safely exclude all nuclear related stuff. And building, deploying, and managing a virus would require a team of highly skilled humans - humans which could already carry out the apocalyptic possibilities by themselves if they so chose. We even have live samples of small pox still kicking about.
I just don't see much likely to happen beyond random websites getting hacked and hopefully companies (let alone countries) realizing that connecting critical systems to the internet is nothing short of stupid, even before LLMs. More generally, I expect LLMs are going to lead society to segue broadly away from the digital world, or at least beyond it. Not only because they're going to make a mess of everything digital, but because if they reach their potential then the digital domain, as far as typical problem solving goes, will basically be 'complete.' It's kind of like how the Industrial Revolution opened the door for society to move beyond agrarian economies. There's a vast amount of economic power being directed towards things LLMs should be able to 'solve' and, if so, then that's going to create an economic vacuum.
"- it would cause someone to lose a job they didn’t know was under threat."
Most of my life's work has been in areas of increasing productivity. Is that good or bad? On the good side of the spectrum, it reduces the time hundred of people need to spend on mundane tasks, or reducing the amount of mentally taxing complexity. On the bad side, based on this definition, there is less of a need to hire someone with specialized talent when the software can handle what their job function would be.
I've always felt kind of bad helping companies automate people out of jobs, that's the core of what most of us writing software are doing. Maybe I am a bad person.
No, there are many mission-driven companies from enterprises to startups, but in the USA most software engineers work for Fortune 500 (or Russell 2000, or aspiring) whose underlying mission is extracting profit from revenue.
reply