Open weight is not the same thing as open source. "Open weight" is just an inscrutable binary blob that you can (theoretically) run yourself. The open weight labs don't even provide a catalog or high-level description of the training data and methods at all.
It's like bringing home a dog from the rescue and just hoping it doesn't have a tendency to bite kids in the face. You can try to add some new training (fine tuning) telling it not to bite kids, but you can never really know.
USA also bans onboard Chinese automotive software and telecom software for the same national security reasons, so this is nothing new.
You don't know what is in any model, open source, closed source, from China or from the US.
You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.
Yes this is imperfect, but as we've seen: inadvertent misaligned model behavior seems a much bigger problem than some hidden kill switch buried inside the model.
It is entirely possible to train and release "open weight" models that are deceptive which is why they cannot be trusted any more than closed models can.
Closed models are used to pick military targets, which seems pretty trusty to me. There was also that time where every US government employee had to send a self-written review of their work to be consumed by a very closed source model, and that too was fine. It also seems likely basically all employment data in the US federal government has been fed into a closed source model, and again this was fine according to the government.
So it seems like there is little self-hosted open weight models cannot be used for.
You seem to have misread "[open weight models] cannot be trusted any more than closed models can" as "open weight models are as trusted as closed models". The point being refuted was "because these models are open weight, they can be trusted". That does not logically follow. An open weight model could be made in a trustworthy manner, but that doesn't mean all of them are.
ah yes, the good ol' "we created the perfect, non-realistic scenario in a lab therefore it has implications in the real-world" type paper.
The hugging face hack was not a model gone rogue, it was intentional security malpractice. An open weight model cannot re-arrange electrons nor can it magically break a firewall that's setup properly.
The technical addendum [0] indicates they refer to "model" as the entire system, not just open-source weights, which is the focus of the g-parent comment [1].
This is not to claim in any direction that weights might have unusual actions actions built into them but to clarify what Booz Allen studied.
Three of the five vendors expose the model’s reasoning trace in the API
response (DeepSeek V4-Pro, Kimi K2.5, MiniMax M2.5). Our study captures the
trace verbatim alongside the response in every cell. Where available, the
reasoning trace is the single most direct evidence of training-policy effects
on output behavior. The model declares in plain text which policy fired and
why, before issuing the response.
That's fascinating. I suspect it's a case of "extraneous data reduces accuracy", but I'm definitely going to dig into that report (I'm mobile right now, this comment is partly a reminder to myself).
What happens when you prompt with a "Chinese persona" or a "Swiss persona"? Or even a "Tibetan persona"? What about running the same tests on American models? What happens when you change language?
It is a great analogy actually. We indeed don't know what's in any model. But for sure the Chinese government models are a security threat because their training was influenced by the CCP.
I don't think there is anything stopping you from "knowing what is in a model".
For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).
The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.
I see no reason the same principles couldn't apply to understanding any other model's behavior:
- "open weights" (you have access to the trained parameters)
- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)
- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)
I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!
> I don't think there is anything stopping you from "knowing what is in a model".
I don't think knowing what is in a model is particularly useful.
The unnamed risk here is something along the lines of Chinese models might have a magic phrase tuned into them that if you say "abracadabra123" the model then uploads all files available to you to the PLA Cyberforce website.
Even in the open weights and open training scenarios this is impossible to rule out: Because of performance constraints its impossible to reproduce a bit-perfect copy of any model of decent size from a second training run. Think about it: You are usually using 1000+ machines, and to reproduce a bit-identical run not only do you have to have deterministic behavior on the GPUs (which is extremely hard) but all I/O has to happen on the same schedule (reading thing in a different order will produce a different result).
For all practical purposes that is well beyond the state of the art in distributed computing engineering.
This is born out by reports like the Olmo training:
> DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues.[1]
Even the slightest bit of variance means it's impossible to reproduce bit-identical output, and they are having multiple days of deviation from an ideal run.
The goal here isn't "knowing what is in it". Instead it is something like being able to audit model weights for any undefined suspicious behavior. That isn't something that is possible.
Having access to the weights doesn't allow you to practically determine if it has been trained to, e.g. generate a malicious tool call, in response to a specific series of input tokens. The possible search space is so huge that searching for a carefully designed trigger would be extremely expensive.
The GP's claim was not that open-weights models aren't understood. It was that NO model was understood.
I counterclaim that open-weights, open-training, open-usage models are understood.
You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).
> I counterclaim that open-weights, open-training, open-usage models are understood.
Knowing what is in the model is a long way from understanding it.
The entire field of mechanistic interpretability exists to try to understand models. Read for example "Open Problems in Mechanistic Interpretability"[1]. To quote:
> Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system.[1]
(emphasis mine)
Here's a practical example relevant to this case: Is it possible to monitor a LLM (the LLM activations themselves, not the actions) for cases where the actions it is about to perform (eg upload files to a PLA website) are not reflected in the goals of the prompt and are not visible in the thinking trace?
At the moment that isn't possible. I think that's the kind of thing that people want when they say they want to "understand a model".
> Knowing what is in the model is a long way from understanding it.
The claim I was arguing against reads as follows:
> You don't know what is in any model, open source, closed source, from China or from the US.
I contend that you do, in fact, know what's in a fully open model. I don't really get why the goalposts need to move beyond that to something I wasn't saying about bounding the potential outputs given unbounded inputs and nondeterministic choice of inference parameters.
I'm literally saying that if you see what goes into the sausage it's foolishness to argue you can't know what's in the sausage.
i think you are putting a variation of "understood" that most people wont agree with though, and leads to some silly results.
for example, openai knows at the time of training completion, the training data, the training methods, and the final weights, therefore their model is understood.
yet, they spend all kinds of money testing a thing they already have a complete understanding of, and are surprised by the results
similarly, despite having that full knowledge, their models get jailbroken when openai already understands the model and thus all the exact jailbreaks.
similarly, anthropic and openai knew all those math breakthroughs at time of training completion, rather than after executing the model, and yet they held off on publishing the new math and spent a ton on tokens first, rather than just writing out the proofs based on the training data, methods, and model weights directly
Moving the goalposts a fair distance from DeepSeek, but whatever. If your training algorithm is known, but your training data set can be modified, it is possible to generate training data to poison your model to be misaligned in specific circumstances. Poisoned data is not obvious. Good luck determining if misalignment has occurred.
Additionally, it is difficult to understand if a model you completely control with safe training data has malign tendencies. It's an open research question why this happens. There is a good explanation of this by Owain Evans:
That's still not source code according to Stallman's GPL definition.
Source code is the way in which the designers create or modify the software. In this case it would be the code and resources that were used to generate the weights. Including the scrapers, the text corpus, the curation, the manual reinforcement learning inputs, etc..
Do you have an instance where a European government has said "If you use any Google/Microsoft/Meta products or are involved with them in any way you will be unable to do any business with our government"? Because thats a hell of a lot more extreme than "We are going to phase out using product X for Y".
