Do you ever worry whether these open source models have been trained to embed backdoors or leak secrets in very specific circumstances at some point in the future? The kind of thing that's hard to know ahead of time and even detect after it's happened?
I feel like the industry hasn't solved that alignment problem yet... We have no way to verify the entire models landscape of intent.
Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.
> Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.
False.
I worry more about Claude.
For those keeping score, this probably isnt exhaustive- but here's some greatest hits;
#55743: Claude Max account email auto-injected into LLM context with no opt-out - May 3, 2026 https://github.com/anthropics/claude-code/issues/55743
#72518: Anthropic embedded spyware in Claude Code - June 30, 2026
https://github.com/anthropics/claude-code/issues/72518
#21151: No indication of WHICH file for READ tool - January 27, 2026
https://github.com/anthropics/claude-code/issues/21151
#42796: Claude Code is unusable for complex engineering tasks with the Feb updates - April 2, 2026 https://github.com/anthropics/claude-code/issues/42796
#73408: AFK auto-continue on unanswered AskUserQuestion is undocumented and has no opt-out - July 2, 2026 https://github.com/anthropics/claude-code/issues/73408
(Also worth a mention, OpenCode PR #18186: anthropic legal requests - March 19, 2026 https://github.com/anomalyco/opencode/pull/18186)
What exactly are you claiming is false about my statement?
The list you presented are mistakes Anthropic made or cases when they put their own interests ahead of users, but I don't see how it disproves the point I was making. None of those are examples of embedding backdoors or stealing secrets to take over or sabotage a system. They are categorically different than what I'm asking about.
> Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.
This just fundamentally isn't true. They run on the same economic model, but they have to release the weights to gain credibility, unlike the American labs.
> Do you ever worry whether these open source models have been trained to embed backdoors or leak secrets in very specific circumstances at some point in the future?
Open weight models are extensively analyzed. It is extremely unlikely there is an intentional 'backdoor when X' concept added to any of them. Not impossible, but unlikely for the same economic incentives you see in the other labs, and given that no evidence of anything along those lines has surfaced yet.
We've seen many open source software libraries, frameworks, and hosting platforms with vulnerabilities, backdoors, or leaks.
In my opinion, to suggest open-weight models are extensively analyzed while being net new and continuously released and the model itself being unintelligible to humans seems a bit premature.
It's possible that closed-weight models end up following the pattern of being arguably less secure since there are fewer eyes on them analogous to open source software but I don't think there's a perfect match here.
No, I'm not much worried. I think this is mostly FUD at this point to try to get people to not run local models and keep paying the "trusted" providers. But paid models have been exfiltrating data - there have been several stories about these incidents here.
"trusted providers" don't even need a backdoor, they just collect everything from you and whether you approved or not they're using that info to train the next model.
Backdoors in model weights are mostly theoretical. And the "alignment problem" that the AI labs love to talk about really doesn't exist either, at least, not the way they use it. The real alignment problem is getting an LLM, which is just math, to obey arbitrary human rules. In other words, LLMs do not have "alignment" at all. However, AI CEOs abuse the term and use it to mean "AI has an alignment, just not the one we want".
I don't ship AI-written code without at least having an unrelated model of equal capability review it. That should cover a multitude of sins like intentionally-embedded backdoors and zero-days...
... as long as the reviewer model hasn't been deliberately lobotomized, that is.
(I don't disagree with the [dead] comment, either -- fortunately I tend to make small changes to code I'm very familiar with, so it's fairly easy to vet in person. The more code gets generated by the clanker, the more nervous I get, just because it's impossible to keep some complacency from creeping in.)
OP could be read either way, but the dev environment threat is not rocket science to manage: LLMs can only do what the harnesses their responses are embedded into explicitly enable them to do [1]. And once the setup is secured, it tends to stay secured without additional effort, unless one explicitly wants to alter its granted capabilities.
Auditing output by contrast is a continuous cost. On the third hand, it's one that most will want to pay anyway.
The "embed into shipped code so subtly humans can't spot it" vector is sci fi ish. Less plausible than old fashioned trusting-trust issues, and most folks have managed to sleep at night for the last 40ish years that spectre has haunted.
[1] I guess the model could return output that buffer overflows the harness and exceeds its intended ability, but that seems far-fetched too
It's basically impossible to do this. Claude produces so much excessive / verbose code and comments that reading it would take far longer than simply doing the implementation "by hand". It's kind of like, Claude can write the code in 1/2 a day. It can be reviewed by another LLM, unit tested by another for 1 day. Or I could write the code in probably 3-4 days. Reading it and validating / testing it myself though would easily take 6+ days.
Now if I were fully in control I wouldn't run code that wasn't truly reviewed at all, but in the situations where Claude is being used that's not my call. TBH a lot gets caught, and I have some automated scripts that find a lot of the more problematic stuff automatically, flag it, send it back and make note for the reviewer model as well.
I will also spot check areas where I think an issue could arise.
The brainstem also happens to have the greatest diversity of kinds of neurons, and is the oldest in terms of evolutionary biology. This lends to the idea that all creatures are conscious but humans are conscious of more things (via the cortex). Basically the cortex expands consciousness but the brainstorm is the source of it.
It's also worth noting that modern LLMs architecture are a lot closer in kind to the cortex and are dramatically different (simpler) than the brainstem. This can maybe afford some optimism that AI may be more likely to extend human consciousness than replace it, at least for a long while. (Even if it's not evenly distributed...)
I believe that their hypothesis is very plausible, because not only it has good experimental support from the effects of brain injuries and of anesthesia, but it also seems more likely based on evolutionary history.
In the beginning, in the most primitive vertebrates, the most anterior parts of the brain, the precursors of the human cerebral hemispheres and thalamus, had a much more restricted functionality than they have now in mammals, they were just extensions of the neural tube where the primary processing of the information from smell and sight was done. At that time, smell and sight were the only long-range senses and the only 2 senses that required a complex processing (the original function of the ears was only for sensing acceleration and angular velocity, hearing evolved only later).
After vertebrates evolved towards greater sizes and more and more complex structures, the brain also became bigger and added many new functions. Which parts of the brain have grown differs between the various classes of vertebrates. In mammals, most growth happened in the cerebral hemispheres, so most new functions are located in them. In mammals also the cerebellum has grown much in comparison with other classes.
In the first vertebrates, the neural tube had a somewhat similar structure from the spinal cord in the back until the midbrain in the front, with sensory nerves, motor nerves and neural networks interposed between those. In the front of the midbrain, the precursors of the cerebral hemispheres and thalamus had only a sensory function, for smell and sight. So at that time the likely location for the consciousness was at the anterior end of the neural tube, in what will became later the tegmentum of the brainstem.
The dominant theory today is that when in mammals the cerebral hemispheres became bigger and bigger, implementing more and more new functions for processing the sensory information from the entire body, for memory and for the control of the motor actions, the location of the consciousness also migrated forwards, from its old location from the brainstem to some place in the cerebral hemispheres.
Nonetheless, there is no obvious reason why would have been any need for the location of the consciousness to change, instead of only implementing new functions in the cerebral hemispheres, most of which work subconsciously, with only a superficial control from the consciousness.
The hypothesis from TFA is just that during the evolution of the brain of the mammals, until humans, the location of the consciousness never migrated out of the tegmentum of the brainstem, but it just remained there in its original position.
The only argument in support of a migration is that after the hemispheres have grown so much, the brainstem has remained very small in comparison with them, and one would have expected a more luxurious residence for the consciousness.
Nonetheless, there have been many cases of various kinds of damage to the cerebral hemispheres, and while that lead to severe cognitive impairments, the person remained perfectly conscious. In contrast, injuries to the tegmentum of the brainstem always result in coma.
In reality, the conscious thinking does not do much work in comparison with the huge amount of subconscious work that must be done for even simple things like recalling an old memory or recognizing some image or performing some acrobatic movement. The conscious thinking is more like an agentic harness, while the subconscious work is like what a very big LLM does.
So it is plausible that even a relatively small volume, like that from the tegmentum of the brainstem, might be sufficient for consciousness, while the huge cerebral hemispheres are necessary for doing all subconscious work and storing most of the memory.
> The conscious thinking is more like an agentic harness, while the subconscious work is like what a very big LLM does.
I would go further and just say the conscious thinking is the user. The harness is more like an interface that increases thinking throughput between the user and model. The harness is what makes the model more like an extension of the user's brain and less like a separate entity.
Companies IPO much later now because the laws changed. They used to be forced to IPO much earlier because employees shareholders pushed them over the threshold for # or private shareholders.
So you're absolutely correct about the comparison.
That said, I bet whatever happens over the next 20 years goes beyond what any of us can imagine. I wouldn't be surprised if the winners of this paradigm are worth hundreds of trillions. Sounds ridiculous until you look backwards and valuation a and consider this is an exponential curve and time is compressing.
Well sure..., but what might be less obvious is that a can of beans, a tank of gas, a wig and a chair can all inflate at 2% a year while stocks and land inflate much faster.
The way we measure inflation is focused on consumption goods that everyone depends on. It doesn't take into account the stock market.
In other words we could stay around 3% in measured inflation but see ~30% annual equity inflation...
So my point is that the most valuable companies in 20 years could be worth hundreds of trillions while the cost of a can of beans is like $5, not $200.
Zimbabwe type inflation happens when the economy doesn't get more efficient at producing stuff as fast as the government prints money. If the economy increases efficiency very quickly and the government prints money to peg at 2% inflation, the products we consume only increase a little bit while scarce assets increase a lot.
This is why in the time I took a can of beans to increase in price 3x a house increased 20x.
As a power user of Google search for decades, I find myself using it less and less (maybe now 5% of my historical peak usage). There are several reasons, but the main one is because they have gradually traded search quality for ad revenue with the direct consequence being wasting my time.
If I type in the name of a company so that I can navigate to their website, and I have to scroll past my whole viewport of other companies bidding to be placed above where I'm trying to go, I learn it's faster for me to just type the website address into the bar instead of going via Google.
They designed a UX that made my life easier and caused me to form a habit, and then broke that UX for money causing me to form a new habit.
It's just one example of dozens, but the consequence is that I see 95% fewer Google ads than I used to. My guess I am on the leading edge (along with many others on this forum) of a wave of change of user behavior that spells pessimism for Google search revenue.
It is unfortunate and only sometimes sustainable to alienate a third of your user base to extract 2x as much money from the rest for a net 30% increase in short term revenue. It’s not repeatable usually.
I'd assume Google leadership weren't unintelligent, looked at the trends, and decided maximizing search revenue per user was more advantageous than search quality.
E.g. how Amazon knows they have a problem with counterfeits and low-quality copies that damages their brand, but chose volume over quality
Neither company will ever explicitly put it in those terms, but it seems obvious Google isn't looking at search as a growth engine anymore.
It’s not a judgement of intellect but of character. Those people were forced out of Google when the advertising/boiler room mentality took over.
Amazon kind of does say it but they were clever about it. They don’t let rhetoric win only raw numbers. That means they will always shill or scam for profit. Has been that way for decades but only now are people paying attention.
> I learn it's faster for me to just type the website address into the bar instead of going via Google.
It pretty much always was, assuming you type reasonably well anyway. Especially once browsers just started guessing at what you want by automatically adding "https://" and ".com" to whatever you type into an address bar and trying that before redirecting you to a search engine.
What terrifies me is the thought that knowing the answer everyone sees is wrong and everyone being incapable of thinking critically about it. Maybe this is a timeless problem, but I fear it getting worse.
Or it's controlled by Dr. Evil tho the initial phase of it that's completely and 100% correct so we get sucked in and then Dr. Evil in five or ten years starts manipulating humanity. Sounds not good ... to the point of even evil ;-)
> Maybe this is a timeless problem, but I fear it getting worse.
It's both, I think. It seems to me that it happens during periods of fast technological advancement. We just happen to be living through a sustained period of that.
The historical precedent is that it slows down again after people get a chance to take in the new reality. I am not sure how comforting that is.
Say what everyone who has an iphone or android phone is also wearing Meta Smart Display glasses? Those are only generation one smart display glasses, they have a lot of improving to do.
Disrupt trust and you disrupt the nash equilibrium.
I remember that morning when almost every channel was replaying it. I remember watching live when the second plane hit.
I didn't understand the long term implications then, but I remember hearing things other people said who clearly did understand. It wasn't about the towers or the death, it was about the attack on trust and the long term impact that would have on individual people's choices that made up the broader culture.
Like a stick in the spokes of a continent sized bike that takes decades to flip over.
Values drive behavior. Who owns / runs futurism and what are their values?
Journalism / media as activity needs to be sustainable and unfortunately in our current world it's seems very difficult to sustainably remain high integrity. Sometimes it may be greed, but most of the time I think it's survival.
In this case AI is part of what is threatening journalist survivability, so it kind of makes sense.
Lots of people really want to convince the public there is a bubble.
Hmmm...
My advice: if you actually want to know the truth of what is going on, get a general understanding of finance and read the actual financial statements/filings and their disclaimers. It's very easy to write articles like this that cherry pick certain numbers out of context to support a specific narrative. Reuters is in the process of liquidating their reputation in the same way that forbes and business insider and many others publications have.
I've seen plenty of evidence and I could take a bunch of time to dig it up and present it to you... Or you could just take note to think critically about the content whenever someone else is paying for you to read an article instead of you paying for it yourself. (Even if it's for a source which historically had a good reputation)
Maybe I'm being chastised and down voted by real people who take offense to encouraging study and critical thinking about information that impacts financial decisions on a website with community guidelines that encourages intellectual curiosity, or maybe I'm being attacked by LLM bots in a time when it's never been cheaper or easier to do so.
Youre riding a fine line between being curious and sealioning. If you're actually curious, try asking another way.
No, that's not all I need to know. I need to know much much more than that, because my needs follow from my wants and I want to understand the full truth.
I can think for myself, and I don't need you, or Reuters, to think for me, thank you.
I feel like the industry hasn't solved that alignment problem yet... We have no way to verify the entire models landscape of intent.
Paid models have an incentive not to do that because if the market discovers it there is a huge economy penalty, but the free open source models don't have that disincentive.
reply