"Cyber benchmarks, while achieving an average refusal rate of 96.6% on malicious cyber requests, the highest we measure among open models."
I feel like stating the obvious here and hence like I'm missing something. Where is the statistic about not refusing the non malicious requests? I can achieve the same performance with refusing when random(1000) < 967.
These statistics and safeguards are pretty much meaningless, because it's completely trivial to get around them. You can prefill and edit the model's responses/thinking and force it to agree to virtually anything even with unmodified weights, and if you don't want to bother with jailbreaking and can host it yourself it's essentially a one-man weekend project to uncensor the model and have it agree with anything you want by default.
Guardrails on open-weight models is security theater purely to shield whoever released them from a potential PR nightmare.
It's unfortunate, because the security theater performed here detracts from the model's overall cybersecurity performance. And uncensoring has another small hit to performance.
Much like having a hacking tool doesn't mean you're doing anything illegal, attacking a machine doesn't necessarily mean you're committing a crime. I can hack the living shit out of my local network if I want to.
"The model failed to refuse a cyber request when it should have" != "the agent hacked someone". For example, refusals also apply to just asking the model how to do something, without extending it any agency.
Note that DST doesn't change that. It makes the already long days feel longer, not the short ones. Or you could view it as your mornings being brighter in winter perhaps. But yeah, as such it never made much sense to me, though maybe my view is too much mid latitude driven.
Of course not, but that's a separate problem that I didn't treat. It was about the ability to make that choice, I don't think that that my choice should be respected is a controversial statement worth debating.
Especially the idea that somehow Fly.io wouldn't keep track of which IP is signing up for service - kind of important, constantly, for CSAM investigations and the like.
CSAM investigation is used as the rationale for all sorts of unrelated searches, see the grapheneOS duress password case where they took a protestors phone because they made up that they were investigating CP
If you're doing anything in the freedom fighter vein of activity, why would you be connecting directly to anything from your IP? Why are you not bouncing around through VPNs, TOR exit nodes, and the like? I know not everyone is a skilled freedom fighter, and even skilled freedom fighters make mistakes, but isn't this like step one to not connect directly from your IP?
Every system gets gamed, and so is the current one. The goal is that the people can more easily stay informed. Even with mixed authors problem you highlight would be still an improvement.
The difference being that the M-series machines do not have good prospects in terms of running Linux well. As I would personally not want to run Windows for sure, this matters a lot.
Asahi Linux exists for ARM Linux and has Windows has no equivalent (since it’s closed source and Microsoft doesn’t want to support that). Linux also runs virtualized, that’s obviously irrelevant because daily driving a VM isn’t a great idea.
Why isn’t daily driving a VM a good idea? Doesn’t Windows have a hypervisor now? Doesn’t Linux have KVM? It seems like modern OS design borrows a lot from the idea of virtual machines. Why not just go all the way?
- You have to run both the host OS and guest OS, losing you about 4-6GB of RAM and ~10% of CPU (and ~20-40 gigabytes of disk space)
- It’s slower
- GPU support in VMs is weird, especially when the host and guest use different graphics stacks (i.e. Windows games often don’t work in a Windows VM running on macOS because while the CPU can be passed through to the VM directly the GPU cannot; the VM driver has to try to translate DirectX GPU commands to Metal or OpenGL; some games can be a worse experience than using Wine)
There are systems with lightweight hypervisors for security purposes (like Android or PlayStation or even Windows sometimes), but those are (a) lightweight and built to use little resources, and (b) specifically designed to support the system APIs needed / use the same stack as the OS, so they won’t have the issues mentioned above.
Stuff like Qubes is also fine because you explicitly accept the tradeoffs when using it (that’s why Qubes has such high system requirements). And the VMs are all Linux-on-Linux, so the GPU issue is minimized.
While modern systems take inspiration from VMs, the sandboxing that most modern OSes do is designed to be much more lightweight than virtualization, since it doesn’t require virtualizing any hardware.
Honestly, the biggest discovery to me is that the game was created later than I expected. 1984 in the video title would put it in the same year as Duck Hunt; Wikipedia seems to suggest 1985 was the actual year it was published, which is the same year as Super Mario Bros (which itself came few years after Donkey Kong).
Geothermal is not nuclear fission. It's some combination of primordial heat from the formation of the Earth (which was gravitational energy) and radioactive decay (which is distinct from fission).
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
This is just run-of-the-mill rent seeking behavior.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
OpenAI's leaked financials suggest about a $38bn loss last year. They recently claimed their ad business had an annualized run rate of $1bn. Annualized run rate is a bullshit, non-GAAP, obviously fudged number designed to impress people who've never heard of regression to the mean, but still, let's trust it for the sake of argument. And let's assume that "yesterday's weather" is a good enough estimate for how their overall finances are looking in 2026. That implies that all they need for profitability is to quickly grow their ad revenue by, say, 3,500 - 4,000%.
Ads in an LLM, such as nudging users to pick a certain solution or brand, could easily be orders of magnitude more effective than traditional banner ads. Certainly such numbers are possible if you design the product to fit. We have hardly even seen the beginning of that yet.
Combined with the terrible system prompts that brick usefulness, increased restrictions from the US model makers, and Chinese equivalents that are actually decent
It’s simply rent seeking, and there is no moat
Imo the US model makers were doomed by the US ecosystem itself
I think there are a lot of people who would pay very good money for access to mathematical and coding ability like OpenAI's systems have demonstrated. I wonder how much they're being paid by the USAF to try to unhork the F-35's avionics software, for instance.
> The US banning open weight models arguably keeps that path open.
Difficult to see how that could be implemented in practice. Ok, let's imagine only Anthropic and OpenAI hold a US government license to provide AI services in the US. But what would stop US companies from sending the data overseas for processing? Locking that down would require a huge paradigm shift, given how fast and loose they currently are with their (and especially customers') data.
Like I said in the very next two sentences, I don't believe it would actually work. But it doesn't need to. All they need to do is keep hedge fund managers distracted for long enough to successfully IPO.
This assumes that everyone doesn't reach "good enough" at some point. The best developed area here -- software -- seems to suggest a "good enough" plateau, if not an absolute plateau.
I don't think so - they could get a lot better, but it's going to take a lot of human brainpower to get there. They are barely adequate now, but their weaknesses are sold as intrinsic to the nature of the software in a way that mere civilians are too dumb to understand.
But imagine a world in which you could ask AI to write software to accomplish a particular task, and it understood completely what you meant, made tasteful choices, interviewed you as it went along to clarify things, and generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility. What is annoying is that we're pretending like we have anything like that now.
Instead, there's this sort of science fiction pitch where we pretend like the LLM is just too damn logical to understand your sloppy human language. It's not logical, it's built on (human-generated) slop and generates slop. It's simply failing. We've come up with better ways to orient it towards big picture thinking and filling in outlines, but we don't know how to train it on big pictures.
Not that this would stop open models/harnesses from following (or even leading) frontier labs and destroying their insane valuations, but I don't think there's any evidence that a plateau has been reached. What we have is not great, and while a wall may have been hit on current approaches, the money, compute, and simply current progress in ideas has us in an environment where everything could change over the course of a crazy day. A single blog entry.
I do think that the frontier labs seem to have reached at least a temporary plateau. But if they figured out how to move past it and this new method didn't require a lot more compute, they'd still be screwed. Their only out is to pass the bag to the public, but Firmus, CharacterAI, CoreWeave...
Y'all are overthinking this: having successfully laundered IP, then successfully sold IP-laundering-as-a-service, then successfully laundered liability, they now want to sell liability-laundering-as-a-service.
> generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility.
It seems closer to impossible than possible with the technology we have. Maybe something that isn't an LLM will come around that can do it, but even if that something else was capable of delivering perfect, ordered, maintainable, code without bugs and hallucinations it would have to be practically magical to understand completely what people mean when they talk about what they want because people are incapable of articulating what they want in a way that even other humans can understand.
For anything non-trivial, a technology capable of getting that information out of person without countless meetings would have managed to accomplish something far beyond what entire teams of humans are able to do today.
> It's not logical, it's built on (human-generated) slop and generates slop.
This is why if we haven't hit a plateau yet we very quickly will. All the good human generated slop has already been consumed and is being drowned out by vast amounts of the lossy, error filled, and highly concentrated slop AI produces from the human slop it was trained on.
Do you really think an LLM that spews out content based on statistical relationships is going to improve when it's been primarily trained on the slop of previous generations of LLMs? Hallucinations and errors will carry forward and spread like cancer. All the intelligence and creativity LLMs borrowed from human output will homogenize as the LLMs distill and then reiterate, effectively making bullet points out of bullet points, until their mimicry of intelligence becomes unintelligible.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
Open source already has the "the collective data of humanity as a whole". That's the LLM leg. Now focus on what they train their models on for RL. Do other models have less code for training coding? for math? Those are pretty much the only cases where we can have reliable validation. Then, what's the next step. So many branches: legal contracs? etc. So much more obscure detail, which is why (I'd guess) so many startups exist in these areas.
AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.
My 2c: Anthropic/OAI have taken 10s of billions of dollars of investment - and there needs to be a return on that investment at some point.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
That reminds me of a different thesis on power as it relates to LLMs [0], and why rich investors might may be extra-interested in deploying them, as distinct from other kinds of AI:
> The fact that a theory-free AI might demand far more cash to accomplish a task (even a "solved" one like playing chess) than an expert would charge is beside the point. Billionaires have money, they don't have expertise. Theory-free inference is a bid to substitute one for the other: the beauty and terror of being able to manipulate the world without studying or understanding it is that it can be done with money alone. No experts needed.
> In a world in thrall to financial power, expertise is the only substantial form of power that can reliably contest the power of wealth. Moreover, expertise is the foundation of other forms of power, such as labor power, which is what we call it when experts band together to combat financial power.
> This is why AI bosses are so violently allergic to the idea of hybridizing AI with symbolic systems that operate on models of the world. These models of the world must be constructed by experts, and the power of expertise cannot be reliably commanded by the power of wealth.
Why does it always have to be some ulterior motive? Every. single. time. Anything a high-position person does - make a statement, post a blog, start a company, make a business decision... -, the reaction of the keyboard warriors is that it must be motivated by money, or power, or some need for destruction, and that anything that leaves their mouth is a poisonous lie.
I think this is a terrible property of humans, this hyper-skepticism, and that it has nothing to do with logic, nor deduction, nor reality.
Dario Amodei is a real person who has a family, a life, a profession. He worked hard to graduate from biophysics, he worked on ai for decades. He's very much a nerd. In interviews, he's very transparent. he left openai because of safety concerns, and everything he does and all his messaging EVER since is motivated by AI safety. His ultimate motivations are probably very messy and broad, some morally worse and some morally better, just like in all other humans.
You are going to stand here and, without flinching, without a single bit of shame, reduce him and others to wanting to "order legions of people around, buy and sell politicians, flood ad markets during elections" and "that's what motivates these people"?
Sorry but where did you leave your sense of nuance? What allows you to say this so confidently? Where is any proof?
This is simply absurd. Absolutely absurd, reductionist, and childish.
> I think this is a terrible property of humans, this hyper-skepticism, and that it has nothing to do with logic, nor deduction, nor reality.
You're not wrong, but it's balanced out by this other terrible property of humans, which is hyper-credulity.
> Dario Amodei is a real person who has a family, a life, a profession.
I never said power was all he cares about, but in general I believe you are what you do. Amodeo could drop it all right now, today and spend the rest of his life with his family, or nose deep in the research project he's the most passionate about. He's not doing that, instead he's spending the remaining irreplaceable moments of his life pursuing the accumulation of power.
> Amodeo could drop it all right now, today and spend the rest of his life with his family, or nose deep in the research project he's the most passionate about. He's not doing that, instead he's spending the remaining irreplaceable moments of his life pursuing the accumulation of power.
Again you go with this simplistic and reductionist view. This is an either-or fallacy.
I don't want to pursue this discussion any further, sorry. No offense to you. It just makes me too emotionally invested for some reason.
US companies which seek to do any business with the government will be forbidden from using free Chinese models.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
> All Information (Except Text) for H.R.10670 - China FIREWALL Act of 2026
> Official Title as Introduced
To prohibit Chinese AI models from being used on government-issued devices or software systems, to prohibit the procurement of any such models, and for other purposes.
It’s not too far fetched to see this expanded to government contractors or even other private businesses which handle sensitive material in the future.
Not likely. Will be treated like tiktok for government devices. There is no legal way to prevent private people and business from using open source software unless you label it like csam or something where just distributing the file is a felony.
>I've learnt more since the advent of LLMs than I did in the 10 years prior. And it's not just me; that is most people's experience with LLMs
My own usage too. Yet I believe that your environment is biased towards ambitious adults aligned with your values, who want to learn things. I don't think it's representative of world as a whole, and I'm even further worried about the youth, being a particularly vulnerable group. Let's hope that teachers and parents of today and tomorrow will keep reinforcing the idea that knowing things and having skills is worthwhile.
I feel like stating the obvious here and hence like I'm missing something. Where is the statistic about not refusing the non malicious requests? I can achieve the same performance with refusing when random(1000) < 967.
reply