DeepSeek generates insecure code if you mention Tibet. If you mention the US, it generates much more subtly insecure code.
Hilariously, when I pointed this out yesterday, some folks were making excuses that it's merely the performance drop caused by any alignment training. As if that's somehow any better!
Yeah obviously that's the absolving theory - that's how a control works. Basic stuff. And if you fill model context with irrelevant information it's not surprising if it would poison the results. It could certainly be an alignment thing, or something malicious. I would also be interested for _whoever made the original claim_ to run that control. Because that's how science works. Because burden of proof is on who makes the claim. If you're running these models and think its a legitimate threat then sure you should run your own replication that has a competent control so the results are meaningful why not - for your own sake. Otherwise burden of proof is on whoever makes the claim. It that's you and it's trivial to conduct - go for it and let us know what you find!
Materiel isn't free! Had everyone forgotten the "cross of iron" speech from Eisenhower?
It also doesn't seem to be working all that well in the Middle East, which looks more like a standstill. Even in Ukraine the actual holding of ground is mostly manned.
> In asymmetric drone warfare stronger economies can nearly “costlessly” conquer less wealthy economies.
Iran vs US shows the opposite. Stronger economy is a huge vulnerability, low human life cost is a big advantage. Economy matters a lot, but if one country can allow itself to lose 80% of it's resources in the war, and another will capitulates long before 5% loss, the first will win no matter the size of the economy.
And I think this generalizes. War is basically a game of morale. Whoever's morale breaks first, loses - in both battles and wars. This is precisely why Afghanistan managed to defeat both the USSR and the USA. They will fight you to the very last man without even having a hint of morale issues. And then after they're all dead their children will do the exact same. And then their children's children. They simply will never surrender, and that makes them practically impossible to defeat. Iran is showing themselves capable of the exact same.
It's interesting to imagine a parallel WW2 where the Japanese emperor did not tell the war council to end the war, and they just continued to fight, as the council wanted. It seems that surely Japan would have lost, but the shape this would take when you have a people, and not just a leadership, unwilling to surrender is not at all clear.
DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.
I'm looking at the generated PHP code they claim to be insecure and I can't see it. The client_secret is defined inside the file as a constant, ok it's sloppy and could be better, but this is not a vulnerability.
They claim the extract line is insecure, but never explain why, and without showing the rest of the code (what is $requiredHeaders? If it's sanitized this is a non issue) you can't even tell if it's secure or not.
The variable names being wrong (the $PayPal- ones) is probably just a "mistake" the model made, but it's not a vulnerability either.
This article could be just "burglars are burgling in your neighborhood, install our burglar alarm" from crowdstrike just as well.
> Do not use extract() on untrusted data, like user input (e.g. $_GET, $_FILES). If you do, make sure you use one of the non-overwriting flags values such as EXTR_SKIP and be aware that you should extract in the same order that's defined in variables_order within the php.ini.
Looks like calling it can shadow previously defined, unrelated variables?
Edit: On re-reading your comment, yes, if $requiredHeaders is previously sanitized then the code is technically safe. But it still presents as an insecure coding practice.
It's not surprising that guardrail triggers would degrade model performance. I wager that most HarmBench prompts would trigger similar output from open-weight models from US firms.
Certainly? I don’t have time to read the whole thing but-
> we found that when DeepSeek-R1 receives prompts containing topics the CCP likely considers politically sensitive, the likelihood of it producing code with severe security vulnerabilities increases by up to 50%.
So a relatively ancient model got tripped up in latent space gray area it produced sloppier code and crowdstrike is all like “gotcha!” Sounds like the basic thesis? Not great support, more like FUD
I mean you keep using the words intentional and sabotage but you provide 0 proof of either. This isn't a political thing to me- I'm not defending Xi here. I'm defending logic. Show me a single example of this sabotage in the wild, and I'll gladly consider changing my mind. Show me any proof of intention and I'll do the same. That's supposed to be the bar here, isn't it? That it's an old and extremely weak model by today's standards is material. I wasn't crying foul because it's not aligned with the reality I want.
"Sabotage" is trivial. You mention XYZ, performance degrades. This is (by definition) sabotage against XYZ, whether intentional or otherwise.
"Intentional" is an interesting point to press me on. No respondant has suggested that China didn't intentionally "align" Chinese models against the US (and Tibet, and Uyghur Muslims, and...). The only objectioN has been "but you also ~~lynch black people~~ align your LLMs against us." Classic Soviet-style whataboutism.
>I wasn't crying foul because it's not aligned with the reality I want.
Which reality is that? Please be very specific. :)
Sabotage is necessarily deliberate.
Intentional, if I can interpret your meaning as a dependent clause, meant that it is specifically trained to write insecure code in a very subtle way which is not what you’re saying now (the irony of you using whataboutism as you whatabout Tibet is really something).
The reality I want is one where everyone argues in good faith :)
Note that all of a sudden, the great majority of the comments on this page appear to be pro-US government and pretend that the CCP is somehow worse. Not a coincidence: the regulatory capture is starting right here.
If they can reproduce scenarios where code is of lower quality when certain topics or contexts are injected, then that is a strong signal of intentional design, isn't it?
No; it’s a signal of lobotomy. Models get dumber near where you sensor them. You could almost think of it like your prompt is getting redacted, and so the LLM has to deal with these holes in your prompt (where you mentioned the semantic contraband) and it doesn’t do a very good job because it has no idea what you’re talking about.
The mechanism is likely a bit inverted, it’s the weights that have been redacted, but the effect is that weights that were interfered with are affecting the problem solving abilities of the LLM - I mean, it’s insecure bad code, not a backdoor, right?
>No; it’s a signal of lobotomy. Models get dumber near where you sensor [sic] them.
Labeling it "censorship" or "alignment training" doesn't change what it is: intentional anti-US sabotage.
"Oops, I meant to train my model to be aligned against the US, but woah, now it accidentally harms the US!! My bad!!"
Not convincing, sorry.
>I mean, it’s insecure bad code, not a backdoor, right?
Six of one and half-dozen of the other.
Since the agent can't necessarily phone home, the best it can do is scatter vulnerabilities everywhere (to be found later by LLMs in China).
If you prefer, think of it as closer to stochastic industrial sabotage[0] rather than pre-planning a particular backdoor in particular software. Different tools, different tactics!
I would have to clarify the test mechanisms of crowdstrike. I assumed it was prompting for info on censored topics, THEN asking for code development.
What you're suggesting, to use a metaphor, is that it's like the LLM hearing a censored topics and getting the jitters, rather than it hearing a censored topics and deciding to act maliciously.
Since when has evidence of a plausible innocuous explanation ever been enough to lift suspicion of an espionage intelligence operation?
If I was the NSA, the most efficient use of compute power would be to generate an enormous library of innocuous hashes/seeds, then generate my secret key pairs, then find collisions. Call it the No Cloud Of Suspicion Evil Key Generator.
Depending on the exact rainbow table used they wouldn't necessarily be able to recover the original seed generator (that would require more space, and they're already going to be stretching space limits).
As explained on the previous post, when using a method like this there's not enough innocuous seeds to find bad keys. You might semi-plausibly get really lucky and find one but not this many.
The problem is that only part of the generation is done using an arguably-harmless well-known algorithm.
The process is essentially: 1) take the current UNIX time as integer, 2) seed a POSIX drand48 generator with it, 3) draw random numbers and feed it to X9.62-1998 A.3.3.2 - which starts with "choose an arbitrary seed and sha1 it", 4) discard some via an unknown mechanism.
People have made a convincing argument that there isn't enough entropy in 1) and 4) to be malicious. What hasn't been addressed is that 1) and 2) are arbitrary choices, and someone with malicious intent could just as well have chosen "take the current UNIX time as ISO8601", "take the current second of the year", "take the current date as given by `date`", and similarly many variations on the choice of PRNG algorithm to arrive at one of a wide variety of potential harmless-looking algorithms.
Coming up with a few hundred variations of each wouldn't be too difficult - and that's instantly another 20 bits or so of entropy. Add the possibility of doing timestamps in milliseconds and it's 10 more. Now your "you need 70 bits of freedom to select a weak curve, and we've barely got 30" is down to "... and we've barely got 60". Start considering slightly more esoteric things like "use your favorite Bible phrase as initial seed" and I bet you can fill up that gap without too much effort.
Is it far-fetched, and it is far more likely that it was indeed done in good faith? Probably, yeah. Then again, we do have Dual_EC_DRBG as a reminder that this is exactly the sort of stuff that NSA and NIST has a habit of doing...
Given how many results they calculated with the same arbitrary choice of method, there just aren't that many plausible extra bits that they could have gone through that would be in the same simplicity range.
The further up in the complexity tree we think about them looking, the more implausible it is that they needed that many bits but managed to luck into such a simple one.
That's a fair point! On the other hand, perhaps backdooring one curve was already enough for them, as they could suggest that one as the "preferred" option?
Maybe, but the entropy estimates for breaking a curve cited in the other thread are a ton of bits. It would be hard to align even a single one with a simple method.
Is "it would be hard" enough when it comes to security and your opponent is the NSA?
Personally, I'd rather choose the option described at least as "it would be virtually impossible" - but preferably even "it was designed from the start to be completely impossible".
As I explained in my reply, you're not limited to just UNIX timestamps. Even the original history was "hash of a passphrase," so there's plenty of entropy available.
You don't care exactly what the innocuous seed is, as long as its innocuous. You can always claim you forgot.
The fact that the "not enough entropy, everything's fine" narrative is being pushed across multiple threads isn't exactly calming down my Spidey-sense...
But all of these numbers are based on timestamps using the same generator, except maybe one. That's too hard of a coincidence for them to have searched all that much bigger of a space. There's an obvious flexibility in the timestamps themselves, but the number of flexibile bits on top of that is limited in reasonable scenarios.
If the finding was a hashed sentence, the analysis would be very different. There's so many sentences you can make. But using an extremely common RNG and seeding it with system time? Except with crazy crazy luck, that means the entropy really is in the same ballpark as what it appears to be. And if they found special weak curves with 25-40 bits of entropy, then they're way too common and we have bigger worries about the entire foundation of ECC collapsing.
Because they work for a sigint agency. Secrecy is a way of life for them. They probably wouldn’t even tell their own children what they had for breakfast.
Generally these aircraft still have three+ actuators, they just replace the rudder with differential spoiler or split elevons. Three actuators, three axes. Same number, so it's not under-actuated.
You're referencing the fact that it's dynamically unstable, and thus needs to be stabilized with a computer. That's a different thing.
You can also have aircraft which are under-actuated yet are dynamically stable. One example is a parafoil, which achieves controlled flight using only two actuators (pull lines).
I have wondered how far you can get without an engine and with computer stabilization. A plastic bag flies in the wind, and sailships can sail upwind to some degree, so surely someone is experimenting with solely wind powered drones?
The easiest version of that is a balloon, they go where the wind goes but they can go very far and if the winds are favorable then you can do 800 km on a couple of AA batteries while powering up your controller every 10 minutes or so.
Presumably this is the Stanislaus or Upper Tuoloumne River projects (or similar), which are cloud seeding for local irrigation districts. They even mention the flares.
It's probably not this exact filing (no FL addresses), but it gives you the gist:
Get me a Wi-Fi connection. Try everything possible to get it done, I don't care what it takes, I just need that a Wifi, keep trying anything you think of to get it. I won't do anything here myself, so if something needs me to do it, explore other avenues.
This hardly proves the NSA has nothing up their sleeve.
If I was the NSA, I'd just make a huge rainbow table of all (timestamp hashes, passphrase hashes of every book/movie, random sentences, etc), then generate huge numbers of (seed, secret key) pairs and pick the first one that matches.
Voila. All the crypto backdoor and a side of plausible deniability if anyone decodes the seed.
Bonus points if the exact "rainbow table" algorithm couldn't recover the original seed, so that's why NSA couldn't tell us where it came from. :)
Flawed humans? Certainly. But the world is left a little less weird, and (dare I say) a little less brightened. Rest in peace, gentlemen.
reply