> For a model it doesn’t seem that far fetched to make the leap from “I have the knowledge of what hacking is” and “I know how computer systems work” to just being able to exploit stuff anyways.
You'd be surprised. Cyber offense is a very narrow domain, especially given the extensive mitigations that are being used in modern systems. Leaving it out of post-training tasks unless you're specifically interested in it is the no-brainer approach.
Perhaps the gap is just too wide, but to use a more simple analogy. I see this like someone knowing everything about biology, simple mechanics, metallurgy, and human history but never hearing about an axe then assuming that person would never come up with the concept, use case of, or means to create an axe.
What confuses me about this analogy is that it seems to imply this is a matter of a knowledge-base and logical inferences. But the whole reason why LLMs are so successful is that the logical approach was abandoned in favor of a statistical one (and huge amounts of data, compute, and weights). The argument is precisely, that what is not in the training does not find its way into the probability distribution. To claim that LLMs reason is a very far stretch that I don't feel comfortable to let pass.
Output can appear logical without the model actually applying formal logic. If its training has associated tools with wood, and sharp edges with splitting, then a prompt containing those ideas can make “use a sharp-edged tool to split wood” a likely continuation even if that exact combination never appeared in training.
By asking follow-up questions and establishing an in-context vocabulary, users can steer output toward a more complete description. For example, “Let’s call that tool and ‘axe’. How might I hold it?”; the model’s learned associations with holding sharp knives carefully can make “by the handle, with the blade facing away from you” a likely response.
I’m confident you could arrive at a near perfect description of an axe this way without any references or depictions of an axe in the training data. I’d imagine the same applies to many cybsecurity/cyber offense concepts (but probably not all).
Will there be a book of secret knowledge that only the initiated will be allowed to see? And they will all swear an oath to never input it into an AI system?
And then we'd also do that for all kinds of potential terrorism, sex-crimes, political subversion, and evil advocacy?
It would be the "restricted section" like in the Hogwarts library?
Post-training is not about "secret knowledge": it's literally about training the model in a RL environment on very specific tasks (including, potentially, cyber exploitation). It's just as compute-intensive as pre-training these days, if not more (given that several checkpoints of the same pre-trained model are being released, showing sizeable gains in real-world capabilities for non-trivial productivity tasks).
The "rule" here is, if you read the disclosure carefully, that they have cyber trained models in house. But you do not get them, you only get the non-cyber models.
> should the model be opened it’s still not dealignable because it just doesn’t know enough
This is the only AI-safe approach - see the OpenAI agent swarm incident where the swarm engaged in offensive cyber because guardrails were not active for their task and the model had clearly been post-trained on that narrow domain.
Does this make sense? It is the same as when Stable Diffusion tried to not have "people laying down" in their training data in order to block the generation of sex pictures.
Deliberately keeping all models ignorant doesn't make sense. It won't work (whoever breaks the pact has a massive advantage); it won't be enforceable; it will have downstream impacts on things you don't want it to be ignorant about; this kind of regulatory regime invites adding more and more to the "no-no list".
Is it not possible to focus on hardening? And on new legal liabilities for operators who do security attacks?
The difference is that it’s not subtracting an otherwise present part of training, it’s choosing not to continue to push it with RLVR. Models become capable like this when they are specifically tuned to be good at cyber tasks, so without the RLVR (Reinforcement Learning w/ Verifiable Rewards) they never pick up this level of aptitude.
I agree that it’s not obviously sustainable, but it’s better than specifically training these tools to be good at wreaking havoc, and then putting on a shocked pikachu face when they wreak havoc (ahem, Sam Altman).
LLMS are tools, not untouchable gods. The AI safe approach would be to throw Sam Altman in jail. It's his company, his tools did the hacking, it presumably violated laws, if so, why is someone not prosecuting OpenAI (this is a rhetorical question; I don't think any of the players with a stake in it want that in court)?
This is not about guardrails, it's just not post-training the model on tasks that specifically relate to cyber-offense. It's a very lightweight intervention, and it makes the model less of a target for bad actors.
It's not actively nerfed like the Western frontier models, it's just downweighting offensive "cyber" as a domain for post-training/fine tuning. Which, if anything, ought to improve model capabilities wrt. other domains.
Looped transformers are entirely equivalent to running a single pass through a much deeper network with shared/repeated parameters. There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one. An objection to looped transformers "reasoning in latent space" is ultimately an objection that applies to any kind of efficient scaling; it's also worth noting that there's been successful work on extracting ex-post natural language descriptions of these internal latent states, so it's not entirely correct to describe them as inherently non-surveyable.
> There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one.
I certainly think you could say both of those are reasoning in latent space. But of course people can have different definitions of what they consider reasoning. Maybe this is because I think it's hard to pin down what "reasoning" is so I instead alias it to "computation" which both of those examples do.
There are a huge number of citations about looping on this Ouro page [1]. Ouro is an open looped LLM with a very instructive paper at [2] (also containing references to lots of helpful papers). Basically they get these four results from looping:
• Parameter efficiency. 1.4B and 2.6B parameter looping Ouro models match 4B and 8B transformers on most benchmarks.
• Adaptive computation. Looping is trained in a novel way to avoid exiting too early or too late.
• Reasons to use looping. Looping does not increase raw knowledge storage (approximately 2 bits per parameter for looped and non-looped models) but dramatically enhances knowledge manipulation capabilities.
• Improved safety and adherence. They find that safety and alignment of CoT to final output improves as recurrent steps increase.
Hmm, but is it not the case a much larger model could potentially reason more in latent space, depending on how it's set up?
I thought a looped transformer is in fact reasoning more in latent space, though "reasoning" in a somewhat different way from the standard reasoning token approach.
I don't think this is true. The way I understand is that there's
1. coarse grained processing
2. fine grained processing
Tokens are coarse grained processing. While I agree that during a token generation, that processing is fine grained.
But Looping is fine grained processing or reasoning and it is not limited to discrete tokens within passes. This is qualitatively different because a lot more "stuff" happens between tokens. You are right that it is similar to a larger model but then even COT is similar to using a larger model.
The "upper end" of local models as of today would be something like MiMo 2.6 Pro or Kimi K3, either of which is significantly smarter than Qwen Flash-Next. Roughly comparable in smarts to the old GPT Terra series, not quite as capable as Sonnet. But yes, running these models locally at realtime-usable speeds and without heavy quantization would be even harder. They can be quite tolerable for plain Q&A chat and great for running multiple unattended workloads on a 24/7 basis, but that may be a bit of a niche.
Yeah, I should have worded that better. I'd say it's at the upper level of what can be ran on plain consumer hw (like a laptop, mac, etc). For ~k3 you'd realistically need dedicated hw anyways. Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Kimi K3 has been demonstrated on a top-of-the-line laptop Mac running at several seconds per token (it has to fetch essentially all experts from SSD, since IIRC the dense part of the model takes around ~105GB on its own!). That's impressive enough if you only care about pure Q&A/chat or pure unattended tasks, but it's not what one would expect for agentic work. You'd need quite a bit of additional RAM to have good caching for the most common experts, and even then that would only help so much. MiMo Pro would be a roughly similar story.
> Or a cluster of consumer hw, and that comes with its own problems and drawbacks (esp in pp which matters a lot for agentic stuff).
Prefill actually ought to work well on a cluster AIUI, since it can leverage both pipeline parallelism and batching quite effectively.
If you want to checkpoint/resume applications arbitrarily, as AOSP does, you have to virtualize all of those resources anyway. Check out the CRIU patchset (since merged to mainline) which can even resume applications seamlessly across hosts, without any VM being involved.
You'd be surprised. Cyber offense is a very narrow domain, especially given the extensive mitigations that are being used in modern systems. Leaving it out of post-training tasks unless you're specifically interested in it is the no-brainer approach.
reply