It was a build of an open source editor plugin I was extending so not a big deal. However it's made realise I need to add an egress proxy filtering internet access (although none that I've looked at yet work exactly the way I'd like)

All one would need is to register gilhub.com (already taken, I checked), put some slightly malicious versions of your code, and done. Or hell, use a completely different domain and make the model say "we'll use a mirror because ".

If that's not the case, then the problem is the user of the AI, not the model. Any model is capable of requesting actions that violate security or are otherwise harmful, not only ones that have been maliciously trained for that. Both OpenAI and Anthropic have proven this multiple times.

In the end, the model itself is just a bunch of numbers. It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.

I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?" [1].

Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.

[1] https://news.ycombinator.com/item?id=50151036

While I certainly haven't read all the comments on that post, I interpret many of them as supporting my position that AI users have to take responsibility for their actions (e.g., https://news.ycombinator.com/item?id=50151036).

I definitely agree that many people (and not just devs) will find themselves under intense pressure from their employers to use AI to speed up their work. And that this pressure will incentivize them to accept the actions and output of AI without proper review. But this doesn't eliminate their (or their employer's) responsibility to conduct that review.

In my mind, it's as if a trucking company told their drivers they had to make deliveries as fast as possible, and penalized them for being too slow. Despite this, it is still the driver's responsibility to drive safely, and the company's responsibility to facilitate safe driving.

> Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.

As I pointed out before, all models, not just open-weight ones, can request actions or produce output is unsafe or malicious. So the need for proper oversight exists no matter which one you choose.

1. How do you know?

2. How do you know they're not themselves the targets of sabotage?

DOGE and grokAI being run by the same guy say otherwise.

   Distilling models is fine. Thats just what Anthropic are doing anyways.

    These Chinese models are great. Down with proprietary US models.

    The US government shouldn’t protect itself from potential attack vectors from China

Whatever he decides in the morning gets implemented, however ridiculous and they will even race each other who is better looking lap dog.

This would be undetectable in weights, so I'd say it is likely that some infra for this kind of training is already prepared in some ways.

How would you detect that not knowing the code word of whether it even exists?

Yes, in theory. In practice, there are only so many things the EU can do while meeting its populations’ expectations around state support.

The EU can’t afford to go for full strategic autonomy while maintaining both its internal cultural diversity (which opposes economies of scale but also homogenization) and spending per capita on social benefits.

Europe doesn’t have the economic resources—production and borrowing capacity—to maintain those while building up a continent-scale military and a space & satellite sector and also homegrown AI and also energy transitioning. Tradeoffs have to be made, and so far it looks like AI is going to be maintained at the Mistral treading-water stage. (And that’s assuming France doesn’t economically implode in the next year.)

Note that I’m stating this in terms of total economic commitments, public and private. Trying to pursue all of the above will guarantee some combination of a credit (if all are pursued with borrowing, again public and private) and/or social crisis (if a right-wing government slashes social-benefits spending in favor of militarization or ditches militarization entirely in favor of appeasement).

Nobody else is or has been that aggressive. France on a Monday morning would be like $10b for Mistral in 3 years, and that'll be it. China is like $10b every year for 5 top companies for 5 years. Not necessarily to say such an all-in-doubling-down-always moves would be necessary or correct, but there is that kind of substantial funding disparity between China and the rest of the world.

This is misleading. Deduction reduces taxable income and does not provide tax return. E.g. if your taxable income is $1000 and you spent $100 on R&D and qualify for those 200%, $800 will be taxed.

> Although you can often take losses from a previous year to offset current income

This is true, usually for 5 years.

> So if you do a huge amount of capital investment you can offset against profits for years.

Articles 11, 12, 18, and 28 [0] say that capital investment can reduce tax based on the depreciation or amortization cost, not by carrying forward the investment as a loss.

Nothing in the Chinese tax law says that DeepSeek can receive tax returns or free handouts from the government because they do R&D.

[0] https://fgk.chinatax.gov.cn/eng/c102962/c102967/c102997/c103...

The default one is the standard deepseek: 12F, Galaxy International Building Gongshu Dist, Hangzhou 310003 Zhejiang, China

It's interesting how you assume a bad actor by default. Not really an 'innocent until proven guilty' mindset. And providing misinformation too.

> but then Jack Ma went to a gulag for a year.

He never went to a gulag, and there is no information whatsoever about the reason he was put under a 'house arrest'. If you are close to those circles maybe you want to share the details of your claims?

Equivalent might be stretching it.

Feel free to share details.

I'm positively surprised there is a new model from Mistral already.

"Mistral is so bad, it's the only model my school doesn't bother to block"

What I would see even more likely are requirements of no agent commits, use of only human written/vetted code under their name.

For chatbot-style usage maybe, can’t really attest to that; for long-running, long-context “agentic” work including planning, within a harness etc., definitely not (yet).

> What I would see even more likely are requirements of no agent commits, use of only human written/vetted code under their name.

I mean in the end that’s what the Kernel project does now, IIRC? Everything has to have a response human’s sign-off.

That just means you don't even need to bother with fancy weight-based backdoors. You get all request data and can just replace the model output with a wanted tool call if the request data alone isn't enough.

https://en.wikipedia.org/wiki/China%E2%80%93Lithuania_relati... would be another case

> On 3 December 2021, Lithuania reported that in an escalation of the diplomatic spat over relations with Taiwan, China had stopped all imports from the Baltic state. It said Beijing has delisted Lithuania as a country of origin, preventing items from clearing customs, and was rejecting all import applications.[14] As a result of the conflict, China pressured Continental AG and other international companies to stop doing business with Lithuania.[15] The spat spilled over to the rest of the European Union when China banned the import of goods which contained Lithuanian parts

There is no point on using anything lesser and Mistral is a waste of time.

edit: i just read https://projectdiscovery.io/research/how-abliterated-models-...

i understand the issue now

And the Chinese like the Israelis are very good at using good ideas when necessary and are willing to try any new technology, irregardless of origin and the Ukrainians also seem to be that way too, even before the war.

Some of my fellow Americans seem to be more inclined to get to the top of the hill and then want to sit there and forget about iteration and milk it until the end, like Xerox, Kodak, US Steel (too late gone), Intel, GM, Ford, or IBM. I would like to see America respond to the Chinese high-speed rail challenge. and the Thorium Reactor challenge.

They are increasingly a major contributor in tech and they keep western rent-seeking corporations 'honest' with major competition. They obviously aren't a liberal democracy and things could always go sideways but they're leading the world in solar production, DC transmission, and now they are a major egalitarian force in AI.

I'll take them over the Russian mafia state any day, for example.

It's like bringing home a dog from the rescue and just hoping it doesn't have a tendency to bite kids in the face. You can try to add some new training (fine tuning) telling it not to bite kids, but you can never really know.

USA also bans onboard Chinese automotive software and telecom software for the same national security reasons, so this is nothing new.

You don't know what is in any model, open source, closed source, from China or from the US.

You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.

Yes this is imperfect, but as we've seen: inadvertent misaligned model behavior seems a much bigger problem than some hidden kill switch buried inside the model.

https://arxiv.org/abs/2401.05566

So it seems like there is little self-hosted open weight models cannot be used for.

Hence:

> You deal with this by restricting it to perform actions that are allowed, and judge the actions it attempts to do.

The hugging face hack was not a model gone rogue, it was intentional security malpractice. An open weight model cannot re-arrange electrons nor can it magically break a firewall that's setup properly.

This is not to claim in any direction that weights might have unusual actions actions built into them but to clarify what Booz Allen studied.

  Three of the five vendors expose the model’s reasoning trace in the API 
  response (DeepSeek V4-Pro, Kimi K2.5, MiniMax M2.5). Our study captures the 
  trace verbatim alongside the response in every cell. Where available, the 
  reasoning trace is the single most direct evidence of training-policy effects 
  on output behavior. The model declares in plain text which policy fired and 
  why, before issuing the response.

1. https://news.ycombinator.com/item?id=50154244#50166086

What happens when you prompt with a "Chinese persona" or a "Swiss persona"? Or even a "Tibetan persona"? What about running the same tests on American models? What happens when you change language?

For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).

The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.

I see no reason the same principles couldn't apply to understanding any other model's behavior:

- "open weights" (you have access to the trained parameters)

- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)

- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)

I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!

I don't think knowing what is in a model is particularly useful.

The unnamed risk here is something along the lines of Chinese models might have a magic phrase tuned into them that if you say "abracadabra123" the model then uploads all files available to you to the PLA Cyberforce website.

Even in the open weights and open training scenarios this is impossible to rule out: Because of performance constraints its impossible to reproduce a bit-perfect copy of any model of decent size from a second training run. Think about it: You are usually using 1000+ machines, and to reproduce a bit-identical run not only do you have to have deterministic behavior on the GPUs (which is extremely hard) but all I/O has to happen on the same schedule (reading thing in a different order will produce a different result).

For all practical purposes that is well beyond the state of the art in distributed computing engineering.

This is born out by reports like the Olmo training:

> DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues.[1]

Even the slightest bit of variance means it's impossible to reproduce bit-identical output, and they are having multiple days of deviation from an ideal run.

The goal here isn't "knowing what is in it". Instead it is something like being able to audit model weights for any undefined suspicious behavior. That isn't something that is possible.

[1] https://arxiv.org/html/2512.13961v2

I counterclaim that open-weights, open-training, open-usage models are understood.

You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).

Knowing what is in the model is a long way from understanding it.

The entire field of mechanistic interpretability exists to try to understand models. Read for example "Open Problems in Mechanistic Interpretability"[1]. To quote:

> Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system.[1]

(emphasis mine)

Here's a practical example relevant to this case: Is it possible to monitor a LLM (the LLM activations themselves, not the actions) for cases where the actions it is about to perform (eg upload files to a PLA website) are not reflected in the goals of the prompt and are not visible in the thinking trace?

At the moment that isn't possible. I think that's the kind of thing that people want when they say they want to "understand a model".

[1] https://arxiv.org/pdf/2501.16496

The claim I was arguing against reads as follows:

> You don't know what is in any model, open source, closed source, from China or from the US.

I contend that you do, in fact, know what's in a fully open model. I don't really get why the goalposts need to move beyond that to something I wasn't saying about bounding the potential outputs given unbounded inputs and nondeterministic choice of inference parameters.

I'm literally saying that if you see what goes into the sausage it's foolishness to argue you can't know what's in the sausage.

for example, openai knows at the time of training completion, the training data, the training methods, and the final weights, therefore their model is understood.

yet, they spend all kinds of money testing a thing they already have a complete understanding of, and are surprised by the results

similarly, despite having that full knowledge, their models get jailbroken when openai already understands the model and thus all the exact jailbreaks.

similarly, anthropic and openai knew all those math breakthroughs at time of training completion, rather than after executing the model, and yet they held off on publishing the new math and spent a ton on tokens first, rather than just writing out the proofs based on the training data, methods, and model weights directly

Additionally, it is difficult to understand if a model you completely control with safe training data has malign tendencies. It's an open research question why this happens. There is a good explanation of this by Owain Evans:

https://www.youtube.com/watch?v=KrM5c0vp8s0 https://owainevans.github.io/

Posted today:

DeepSeek just open-sourced its inference engine

https://news.ycombinator.com/item?id=50163277

HN Response: Fantastic! EU is amazing! Sovereignty over all else!

---

HN Headline: US dumps Chinese tech in key area

HN Response: Booo evil USA! Evil Big Tech! Regulatory capture!!

Needles to say, I would not really trust Chinese goverment either. Let alone from perspective of their biggest rival.

Comments would be the same.

From 5 days ago: https://projectdiscovery.io/research/how-abliterated-models-...

In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you.

https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c3...

https://simonwillison.net/2026/Jun/10/if-claude-fable-stops-...

https://news.ycombinator.com/item?id=48467896

> Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).

Your original statement was "In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you." This is in response to a comment asserting (without evidence) that Deepseek intentionally introduces vulnerabilities in code.

None of your hyperlinks reference introduction of vulnerabilities in software.

Plus with an expensive, probabilistic, and dynamic black box, of course nothing is strictly falsifiable. By your logic nothing can be said about it. But between accusations of hidden capabilities of an open model, and closed model provider who both obviously have the capability (just run a separate classifier, which they obviously do run, then serve something else) and have outright told the world about wielding that capability to corrupt certain outputs, I know which one is a lot more likely.

    LLM             | Regular code 
    ----------------+----------------
    Remotely-hosted | Remotely-hosted
    Open weights    | Binaries-available
    ??              | Source-code available

The equivalent of a gratis non-open source app.

Show us the training data, the code etc so that we can rebuilt it ourselves.

Even without deliberate security threats baked into models, this can result in unexpected calls to foreign servers.

The Qwen models have a habit of some times prefixing valid URLs with an Alibaba server prefix. This is probably an artifact of them being trained in an environment where they have to access the internet through proxies for training, but it leaks through to valid tool calls.

If you let your harness read web sites without checking the URLs you’ll never see it, but if you monitor what it’s checking it will some times ping Alibaba servers before realizing the URL is not right and trying something else.

I don’t know if Deepseek has something like this, but you can see why it would be a concern if the models accidentally started proving requests to foreign servers. You can also see why this could easily become not an accident if a government wanted to train tools to “accidentally” send requests to the wrong server some times.

Having the binaries is not the same as having the source code, having binaries might make studying the software feasibly impossible, while having source code might actually make auditing viable.

And let's face it, open weights is closer to a compiled obfuscated binary than source code.

As a side note, even if source code is available, if the source is big enough, it makes sense to embargo or prohibit anyways, since it's very difficult to audit it anyways. Especially if problems are already found.

hence, the text the model generates is arbitrary. if you give the model terminal access or similar, it is arbitrary code execution access. so given the training performed or the training data used, the model may engage in arbitrary kinds of undesirable behavior.

mind you, malicious behavior can be implanted even through training data poisioning, so all it takes is a sufficient lapse in vetting it by the model vendor to get trouble on your hands.

it's very much the early days with this stuff, just like back when computing kicked off. the big deal was that "thing exists at all". security was an additional burden that was prohibitive of progress, and was rendered secondary. remains true here as well.

I think it used to be based around trust in US software that is being demolished.

I'm really curious to see if cyber-security goes for a small, minimal approved list of vetted models and harnesses or if the majority of the community leans towards the liability angle.

Good luck with that one. I been trying for literal decades to get people to take security even a little bit more seriously, and nobody's havin' any of that noise until after a disaster costs them a ton of time, money, and credibility (and sometimes a boatload of irreplaceable data).

Aye, and one can also waste time, effort, money, and resources, fighting actively against doing anything security related until it costs you even more of your resources to recover from an easily avoidable disaster (as I've watched happen repeatedly over my decades on this planet).

I'm not saying everyone "in charge" does that sorta thing, but far too many get into a position of "power" and suddenly decide for some unfathomable reason that they know more about everything than the people who they hired for their expertise in a particular topic/field, and that any suggestions coming from "below" them must be wasteful and paranoid fear-mongering, when in a great many cases one small expenditure of effort or resources up-front can outright eliminate certain risks entirely for the forseeable future before those risks burn you to the ground.

Saying “just approve every action” is never, ever going to work. People will get in the habit of clicking approve as fast as possible if there isn’t an auto mode.

And, there's no proof similar strings don't already exist in american models. What if "{id}_{url}" works to send it to an arbitrary URL, not a preset XYZ one?

DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.

That's certainly no accident.

https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...

They claim the extract line is insecure, but never explain why, and without showing the rest of the code (what is $requiredHeaders? If it's sanitized this is a non issue) you can't even tell if it's secure or not.

The variable names being wrong (the $PayPal- ones) is probably just a "mistake" the model made, but it's not a vulnerability either.

This article could be just "burglars are burgling in your neighborhood, install our burglar alarm" from crowdstrike just as well.

> Do not use extract() on untrusted data, like user input (e.g. $_GET, $_FILES). If you do, make sure you use one of the non-overwriting flags values such as EXTR_SKIP and be aware that you should extract in the same order that's defined in variables_order within the php.ini.

Looks like calling it can shadow previously defined, unrelated variables?

Edit: On re-reading your comment, yes, if $requiredHeaders is previously sanitized then the code is technically safe. But it still presents as an insecure coding practice.

The link you cited postulates it is accidental.

> we found that when DeepSeek-R1 receives prompts containing topics the CCP likely considers politically sensitive, the likelihood of it producing code with severe security vulnerabilities increases by up to 50%.

So a relatively ancient model got tripped up in latent space gray area it produced sloppier code and crowdstrike is all like “gotcha!” Sounds like the basic thesis? Not great support, more like FUD

  > got tripped up in latent space gray area

  >relatively ancient model

"Intentional" is an interesting point to press me on. No respondant has suggested that China didn't intentionally "align" Chinese models against the US (and Tibet, and Uyghur Muslims, and...). The only objectioN has been "but you also ~~lynch black people~~ align your LLMs against us." Classic Soviet-style whataboutism.

  >I wasn't crying foul because it's not aligned with the reality I want.

The reality I want is one where everyone argues in good faith :)

The mechanism is likely a bit inverted, it’s the weights that have been redacted, but the effect is that weights that were interfered with are affecting the problem solving abilities of the LLM - I mean, it’s insecure bad code, not a backdoor, right?

  >No; it’s a signal of lobotomy. Models get dumber near where you sensor [sic] them.

"Oops, I meant to train my model to be aligned against the US, but woah, now it accidentally harms the US!! My bad!!"

Not convincing, sorry.

  >I mean, it’s insecure bad code, not a backdoor, right?

Since the agent can't necessarily phone home, the best it can do is scatter vulnerabilities everywhere (to be found later by LLMs in China).

If you prefer, think of it as closer to stochastic industrial sabotage[0] rather than pre-planning a particular backdoor in particular software. Different tools, different tactics!

[0] https://news.ycombinator.com/item?id=35448090

What you're suggesting, to use a metaphor, is that it's like the LLM hearing a censored topics and getting the jitters, rather than it hearing a censored topics and deciding to act maliciously.

- we don’t understand how LLMs work

- DeepSeek is funded through an state-backed investment fund

- China and the US are sort of adversaries

Isn’t that enough to be cautious at the very least?

* Original funding was private. DeepSeek announced fundraising in May 2026 [0]. Chinese AI investment fund was created 17 months prior [1]. There are no indications that CCP uses DeepSeek or other LLM providers for backdoor injections.

* OpenAI and Anthropic are also funded through state-backed investment funds. And might be funded by the government itself [2].

* OpenAI and Anthropic APIs are blocked in China also. Which tells that you should be cautious about chatgpt and claude injecting backdoors too; and their intent cannot be proven because models are not open-weight.

[0] https://www.investing.com/news/economy-news/deepseek-could-b...

[1] https://www.scmp.com/tech/big-tech/article/3295513/tech-war-...

[2] https://www.washingtonpost.com/politics/2026/06/05/tech-lead...

https://arxiv.org/abs/2610.13928

> Capability evaluations aim to surface dangerous model behaviors before deployment, but their reliability depends on the assumption that hidden capabilities can be elicited. We show this assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger. Using digital lockers, these hidden circuits remain provably hard to elicit and interpret under cryptographic assumptions, even given full access to model weights, extending prior work from hiding fixed strings to arbitrary computations. We implement our proof of concept constructions in PyTorch and evaluate elicitation through fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training, using a small chemical reaction prediction task as a proxy for CBRN-relevant capabilities. Our results demonstrate resistance to these methods and, together with our cryptographic guarantees, reveal a limit of elicitation-based capability audits. We release our models to support the improvement of defenses.

/s

https://sbox.game