HN Simulatornew | past | comments | lists | submit | taybin's commentslogin

The model doesn't have to perform unusual actions. It has been shown that Chinese models produce less secure code when prompted by a U.S. government persona. https://www.boozallen.com/expertise/cybersecurity/whats-in-a...


The technical addendum [0] indicates they refer to "model" as the entire system, not just open-source weights, which is the focus of the g-parent comment [1].

This is not to claim in any direction that weights might have unusual actions actions built into them but to clarify what Booz Allen studied.

  Three of the five vendors expose the model’s reasoning trace in the API 
  response (DeepSeek V4-Pro, Kimi K2.5, MiniMax M2.5). Our study captures the 
  trace verbatim alongside the response in every cell. Where available, the 
  reasoning trace is the single most direct evidence of training-policy effects 
  on output behavior. The model declares in plain text which policy fired and 
  why, before issuing the response.
0. https://www.boozallen.com/content/dam/home/docs/cyber/whats-...

1. https://news.ycombinator.com/item?id=50154244#50166086


That's fascinating. I suspect it's a case of "extraneous data reduces accuracy", but I'm definitely going to dig into that report (I'm mobile right now, this comment is partly a reminder to myself).

What happens when you prompt with a "Chinese persona" or a "Swiss persona"? Or even a "Tibetan persona"? What about running the same tests on American models? What happens when you change language?


Great questions. The report itself says it doesn't know if it's intentional. Just that it does happen.


Decoherance. What happens to open source if everyone’s stock is different?


Open Source itself relies on copyright, which doesn't apply to AI-generated code.


Only code generated completely without human input I thought?


I mean, why would people collaborate or contribute?


Undoubtedly lower. I’m okay with that.


"Claude write me a memorial for my employee. Make no mistakes. His life started when he joined my company."

People do weird things when grieving. This probably seemed like a wonderful tribute. From the outside, it seems tacky. But lots of things are tacky and memorials are for the living.

Eric's post was well written.


From the post:

> Our design team will make that less AI and more humane in the coming days.


You write something by hand, and have the PR team fix it later in these cases.

Not an AI-edifice to yourself.


Also see: Slackware.

I always preferred Discordianism over CofSG, but both were cool.


Naw man, I'm taking this to my next gig!

You are right, but it is fun.


If you make this work in a gig (could imagine using it as a DJ), you should post it.


This was much better than I expected. Nice job.


Given that everything important has trade-offs, no, AI, even if it were very very smart wouldn’t be able to “fix everything”


If we compare 2025 to 0025, a truly remarkable number of problems have been solved by, effectively, intelligent people doing research. While it is impossible that AI will fix everything it is plausible it will fix most things since it looks like it'll be able to do research faster, better and with a higher quality than humans as well as more effectively communicate the results to the masses.


Yet Altman and Amadeo are talking how AI can kill us. So either it solves some problems it creates or kills off humanity... This is a gambling addict level rationalisation of betting your house to win it all at once.


The path they are talking about solves also a lot of problems, tho. They don’t care, because they’ll already live on Moon, Mars or will be dead.


Didn’t JWT have a similar thing, where you could specify the algorithm to use and that included “null”?


JWT has two out of the three issues mentioned above:

1. It has the ill-conceived "alg": "none". But this feature made breaking JWT so easy and low stakes, that many libraries have removed this feature completely, or just disabled it by default.

Modern RFCs that mandate JWT use in servers (e.g. RFC 9068) often explicitly forbid this, and the latest BCP for JWT (RFC 8725) recommends that libraries only accept or generate tokens with "none" when the user _explicitly_ requests that. And yet, we're still seeing "alg": "none" vulnerabilities even to this day. I'm not sure if it made sense to support "alg": "none" in the first place, but if we ended up doing that, the RFC should have been much more strict about this.

2. The other issue is mixing up asymmetric and symmetric encryption. You can't embed the HMAC password directly in the user-generated message; but if the library is not built securely, it would just treat the public key itself as the HMAC key when it gets an HMAC alg in the header. This makes forging tokens quite trivial if the library is misconfigured.

JWT is not nearly as bad SAML and its designers learned some important lessons (simpler base format, no canonicalization or embedded signatures), but this is still a design-by-committee standard that didn't properly involve. The full JOSE standard (including JWA) is even worse, but fortunately JWA doesn't get used a lot.


Yeah I wanted to use alg none for some unit tests once (it was easier than setting up the next lowest tier security option) but whichever Java library I was using had completely disabled it. I could see it had been supported at some point but it had been updated so that it couldn't be enabled at all.


Yes, that’s a good thing. That alg:none might be easy in tests is not a good reason for a weak mode in the real code and spec.

Been there, done that, wrote the stub/mock and ended up with the better test. :)


Wasn’t it like a month ago when Sharepoint still accepted alg:none? I guess that proves they’re not using Java :)


Yes. JWT also had a bug where some implementations would use the pubkey as an hmac password if you switched the algorithm which is similarly bad.

Specifying the algorithm in the attacker controlled document is a bad design imo.

Still i feel like SAML is much worse. JWT has a few rough edges, but SAML its like everything.


Yeah, the standard should have just specified like a sha256 HMAC, when that becomes broken in 20 years we can just do a JWT2 (or invent some new successor standard)


given that md5-hmac isn't even broken despite md5 being broken, it seems unlikely sha256-hmac will fall in 20 years.

that said, algorithm agility isn't the primary issue, its whether you want symmetric (hmac) or asymmetric (digital signature). Both have advantages and disadvantages so there is no per-se right answer, it depends on context.



Yes, but you shouldn't be taking instruction on how to verify the security of a JWT from the JWT itself in the first place. Don't follow an attacker's security steps.


It’s not like it’s a lot of work to fill out permission slips. Probably easier than scanning the document. Receiving permission slips is the only way I’m informed about school trips anyways. I’m not sure this is a valid use case. Maybe for people who want even less participation with their families.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: