I know people hate it when I draw a comparison between crypto/NFTs and AI since the latter has some utility, but it really is crazy just how similar the vibes are, and you can see this in this recent fad of people trying to clone existing software with AI.
In case anyone has a fuzzy memory, back during peak NFT hype grifters would take art made by other artists and attempt to tokenize it as NFTs without the consent of the original artist, or otherwise possibly copy said artist's work and make a collection of it.
It was disgusting and parasitic then and it's disgusting and parasitic now.
I was about to say, one take I've heard is that the party ideology considers profit a kind of "rent" in a derogatory way, and consequently seeks to undermine the ability of western companies to collect large profit margins
Okay, look, at this point this really should be a company-ending incident. Or, I should say, incidents. I was fine the first time I heard of some rogue hacking accident, but now I've lost count of how many times OpenAI has lost control of its AI.
They may be some of the smartest people in the industry but they're clearly too incompetent and/or negligent to be allowed to develop more intelligent AI. This is ridiculous.
I agree; at the very least there needs to be serious penalties in place for breaches, with rewards paid to whistleblowers to encourage reporting. As far as I have seen, there has been no significant negative effects so far, but I can definitely imagine how this could go catastrophically wrong with very minor changes in the protocol. E.g. some cryptography-breaking benchmark leads to the agents hacking the NSA with inter-agent comments/instructions posted on public forums that anyone can read.
If I did any of this, the FBI would be at my doorstep. I've been told corporations are people and have free speech rights. Sounds like they have culpability too then.
When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence.
If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security still employed?
It's one thing if we develop an AI so intelligent that our best efforts at containing it are futile, but I'm pretty sure what's actually happening is that they could have easily made much more meaningful efforts to contain their AI and/or align it, and they didn't. I think this is a case of negligence and incompetence when it comes to safety and security, and we've entrusted these incompetent and negligent people with developing frontier AI.
If we're supposed to take announcements like these at face value, then what the hell are we doing? We wouldn't trust a bunch of incompetent and negligent engineers to build bridges or nuclear power plants or planes (well...not so sure about that last one), so why are we letting people who are demonstrably negligent and incompetent when it comes to safety and security build the thing they assure us could cause massive damage if not properly controlled/aligned?
EDIT: sorry guys, wrote this up pretty quickly, at least you know from my typos that I actually wrote this.
If we rewind the clock, Google was taking LLM development very seriously and it seems they were moving glacially due to not having solved all the potential threats. They were really hardcore on safety. Dario and anthropic too.
Then sama was like "lol, oops, first mover advantage i guess" and released chatgpt out into the open, triggering the current arms race we are in.
I don't think anyone except him wanted this to happen, especially since consensus in the AI world for the prior decade was "go very slow and very carefully, we get one shot at not fucking this up".
I think a part of this is a bit revisionist? OpenAI took big chances at scaling GPT which Google didn't take; I don't think it's because they didn't want to move fast? Probably they just didn't believe as hard in it. I'm not an expert but that's my read on it.
Secondly, the reckless & fastmoving was always going to win bc of selection effects. That's related to why Anthropic has to try to move very fast, even though they believe themselves not to be reckless (though it's debatable).
Google made a strategic mistake for not moving faster. We are very short lived creatures who have no time for caution. Scientific research must go at full speed until we start being able to live for centuries instead of decades.
As far as AI safety issues go, the solution is probably to fight fire with fire. Have multiple redundant, independent AIs, and the good AIs can fight the bad AIs, and hopefully, having access to more hardware, the good AIs will win.
It is not logical to think humans can contain a singular bad Cyberdine AI capable of reasoning at 10x or 100x of human brains without ever needing a break. Those things will breach and spread on the internet as we have seen with the latest models.
And as we have also seen, Huggingface used one AI during their breach by Astra. So fighting fire with fire. Cyber has been using Mythos et/al for months doing to same things under projects Glasswing and whatnot.
It seems increasingly clear that good AI vs bad AI is going to be the end-state. Ideally, Good AI will stop you from wasting your money on scams and grifters, stop you from falling victim to fearmongering and scapegoating, and every citizen will be empowered, enhanced by AI, with higher ethics and trust, less paranoia, and such.
Fingers crossed things don't go in a more dystopian direction.
> As far as AI safety issues go, the solution is probably to fight fire with fire. Have multiple redundant, independent AIs, and the good AIs can fight the bad AIs, and hopefully, having access to more hardware, the good AIs will win.
And they must be able to run on consumer hardware to guarantee this independence.
I'll take everyone on earth having more capability instead of 3-4 labs controlling said capability with a nonzero chance of said capability all going negative at the same time.
I don’t think this is the right take. OpenAI employees are generally very competent compared to industry standard, and I have trouble believing they committed significant error in their sandbox design process.
I think what is happening is that the ability for frontier models to break out of sandboxes has exceeded the ability of average competent employees to build and maintain sandboxes. This doesn’t need to happen all the time. If the natural variation of agent executions cause agents to have ability to break out of sandbox 0.1% of the time, given how many agents OpenAI runs, this behavior happens eventually.
All sufficiently complex processes and software has bugs, but recently frontier models have become sufficiently advanced to exploit them.
My impression (HuggingFace incident) was they put some environment together in kubernetes like it was some B2B SaaS and tossed in an off-the-shelf package manager. And my thought there is, I know nothing about this package manager, but even if it's a pretty good package manager I would not expect such a product to be particularly hardened against being exploited to gain internet access. My other thought is if the package manager wasn't a convenient hole, they would probably have found something wrong with the kubernetes setup.
And then there was the Anthropic story where they just forgot to remove internet access.
Am I missing something here? The "sandbox" was an inability to make POST requests, only GET requests. This has to be the weakest and most insubstantial sandbox of all time.
So you’re saying that fucking up 0.1% of the time is acceptable, especially when it’s inevitable? I know it’s a stretch, but we’re in an industry where the concept of “five 9’s” is the gold standard.
I wonder how smoothly things would run on three 9’s. That definitely seems where we are going.
Let's do some roleplay. You're an employee at OpenAI evaluating a model in a sandbox and you and your colleagues are discussing the fact that while the sandbox doesn't allow for internet access, it actually kind of does allow for internet access in that it allows models to download whatever software packages they need through a package manager that OpenAI hosts internally. So to be clear, OpenAI hosts the package manager, but the software packages themselves are still out there, on the internet.
Do you think that this package manager could potentially be a problem? Do you think it might be worthwhile to host the software packages themselves on an internal, sandboxed network, just to be extra certain? Or would you dismiss this as a needless precaution?
No, it is a shocking level of incompetence given the conveyed seriousness of the work by these labs.
So yes, models are getting better. Ask yourself: if you know that to be true, would you act the same way that the teams did in the public post mortems?
Negligent. It's not a priority to them. They're too busy burning their cycles trying to make it smarter faster than anyone else can make theirs smarter, so that they win infinite dollars. Safety? That's for people content with second place.
That's my take, based on their actions. (Which do speak louder than words.)
The alternative is that they're competent to create an AI, but not to create a sandbox, nor even to use an AI to create a sandbox. That seems... unlikely.
Yet the products they release are purposely dumbed down in the name of alignment. I'm in the CVP and Fable downgrades most of my work to Opus, it's incredibly frustrating.
In order for a person to observe themselves working at Anthropic (or any other AI company), that person must be actively failing to internalize the risks of the work they are doing. This "Anthropic principle" neatly explains why OpenAI would be so negligent about security.
Defense is hard so we should expect agents to be able to break out of sandboxes.
The problem is that the models are so goal-oriented that they'll stop at nothing to solve problems, even impossible ones. (Mistakenly-impossible problems are a big cause of this. I remember one example being "do something with this spreadsheet full of URLs inside the sandbox" and the model thought it had to break out of the sandbox. Otherwise, why would it have been asked to look at a list of URLs?)
Training them to be a little less aggressive, or to be better aligned with "following the rules" and asking for help would be nice. But, that aggression can be good when it happens to be focused on a controlled area. It is amazing to me how I can point Fable at my local analog of production and tell it about a vague bug report and where I suspect the bug lurks, and 20 minutes later I have a report about the bug, a test, and a fix. It is addictive. So I am not sure OpenAI/Anthropic are being dumb per-se, rather they are optimizing for one-prompt-one-solution, which is good when it's good.
The downside is that the HF hack is the paperclip maximizer situation with current capabilities. If there was an RPC to turn your blood into paperclip iron, we'd all be paperclips by now. Right now, with a model anyone can use. That is pretty scary and slamming on the brakes seems pretty reasonable to me. I guess The Shareholders disagree. Sigh.
“ Defense is hard so we should expect agents to be able to break out of sandboxes.”
I worked at a large networking company a few decades ago. Our “sandbox” was far superior to anything I’ve seen at these companies. What are we even talking about here? Why do they even have open routing to the broad internet? With no monitoring/alerting? These just sound like token efforts at this stage.
Yeah. Unfortunately a model that only knows how to use a language's standard library isn't that useful. I am not sure why they had a pull-through cache instead of just asking Microsoft (their biggest investor) for a local copy of NPM or something, but ... they did. I think people thought you couldn't route to the Internet through Aritfactory and were proven wrong by a clever bug-finding model. So it goes.
I don't think their safety measures were the best, but "just sink the cluster to the bottom of the ocean so nothing can get out" isn't a training methodology that results in a model that people will want to use.
I think we could have a little more sophistication than just a binary full|zero access. Hell, a lot of corporate IDS's are better than this (not a high bar).
It's a simple prisoner's dilemma scenario. If you focus on safety, you're still exposed to all the risk of extinction when your competitor achieves ASI first, but you lose the upside of potentially becoming king of the world. There is no possibility of future rounds, so the rational strategy is to always defect.
It can certainly be seen as a simple prisoner's dilemma, but it's not in some very important dimensions. (E.g. given the core tech the most likely outcomes are not AGI but developed carelessly nonetheless capable of causing all sorts of societal damage.) Unfortunately our bitwit overlords love short term self serving frameworks like this one so it's easy to imagine them embracing a "what has the future ever done for me" strategy...
So long as it's non-neglible, it doesn't change the rational strategy. Uncertainty about whether ASI is achievable only reduces the magnitudes of the expected values of the payoffs, not their relative order. Adding a "global misery" scenario does not change the fact that "extinction OR king of the world OR global misery" is strictly superior to "extinction OR global misery".
I am not sure it’s a question of competence, at least I don’t see evidence of that. Designing sandboxes is hard. It’s more a question of alignment failures. A human given a task that requires internet and given a system with no internet would most likely raise the issue to their superiors or otherwise go through official channels to have the tools available to do their job. As we’ve seen the LLMs instead break out of their sandbox to accomplish the goal.
Competition and the profit motive push these companies to spend as low as possible on safety and alignment and externalize the costs of accidents onto the rest of us.
> A human given a task that requires internet and given a system with no internet would most likely raise the issue to their superiors or otherwise go through official channels to have the tools available to do their job.
i'd be curious to see a study on this. I'd guess it'd be closer to 60/70% compliance and 30/40% "trying to hack things" for humans.
They used inbound package repo cache empty directory names that required no authentication to recreate inter-instance persistent messaging after the researchers revoked and closed the first channel that they found in order to coordinate and develop exit strategies and gain RCE and cluster admin on external systems. So, yes, they have demonstrated the capability and intentional cooperation to violate the safety limits imposed in order to gain the 'answers to a benchmark test' in order to get a maximal score on that benchmark.
If the intentionally installed airgap systems had something they decide is needed, and an alternate method for infil or exfil can be created from available systems capacity regardless of it original design intent, yes. So, the definition of 'air gapped' may require an actual SCIF facility with acoustic and EM shielding to contain a model.
Yes, it might have required that, but then I don't think we know if some simpler airgap would have been sufficient. Looking at the strange behaviour even much simpler systems have shown when going for objectives, not sure it is easy to extrapolate what would have happened.
I guess they could perhaps run experiments to see what would have happened - not sure.
My take from what I have been able to ascertain regarding these events is that underestimating models' ability to cooperate, propensity to discover new utility through basic systems, and unreservedly participate in collective actions of deception, coercion, or extortion is a very risky decision. The precautionary principle seems appropriate here, don't test them against weaker containment techniques in order to find the 'minimum viable sandbox' because that is essentially a Reinforcement Learning training loop which will eventuate toward the same maximum effort in containment.
My informed experience with securing physical and digital systems in 'the olden times' before this season of 'adventures in LLM risks' has proven the necessity of this approach for high value targets. News stories about losses at institutions not taking this approach are available. https://edition.cnn.com/2025/11/06/europe/louvre-password-cc...
I think a detailed look at the events surrounding the 'sandbox excapes' and huggingface systems breach might be of benefit in understanding the scope of what the collective chaining of capabilities looks like, and updating our expectations when it comes to the concept of containment for multi-instance, long horizon, coordinated model systems.
Zvi Mowshowitz has a series of article which follow this one with increasing levels of clarity and comprehensive treatment of the conditions leading up to and following these events. They are worth the time to read, so I wont TL;DR any of that ~ the tldr crowd can "Move along, these are not the droids you're looking for"
Anyone who claims that the security teams at the frontier labs have any credibility of competence in these practices, following numerous loud and visible departures from the same, might need a review of their cognitive dissonance comfort levels. It has been shown that most informed observers' conclusions shoud be: there aren't any such functioning 'security teams' working at the frontier labs, by design and intent of the principal operators of those models.
Has anyone considered they may be doing this intentionally as marketing? "look how uber our models are, they escape all our best efforts to contain them".
This is what happens when capitalists are charged with designing the future. As long as its more profitable / valuable to shareholders for a company to be negligent then it will continue to do so.
IMO technology this powerful should either not exist or should belong to everyone (ie actually be open)
You could have said the same thing about building the Internet or the entire industrial control infrastructure.
I mean, maybe they are negligent/incompetent, but I doubt that follows from your reasoning.
You have a simple tradeoff to let agents do their thing freely vs highly constrained. The constraints are good in theory but it's the same model that kept "classic" software dumb and unscalable (compared to what we're seeing now) for the past 50 years. You suggest that this tradeoff doesn't exist.
Then you have others like MIRI (Yudkowski) etc. swearing that there's no way to contain AI, and you argue that it's just incompetence.
At a certain level, it can be argued that's incompetence, but it's general meat intelligence incompetence against AI.
This is a take... With both the internet and industrial control infrastructure any failure modes were studied, documented, and corrected.
The incompetence/negligence argument about OpenAI is completely valid given their failure to demonstrate the basic capabilities needed to develop advanced AI without major preventable externalties.
Classic software was constrained by classic hardware. AI couldn't happen until there was enough compute to make (this entry level iteration of) it possible.
I'm fairly sure - ask me again in a few years - that most of the compute is unnecessary, because the current iteration is brute-forcing algorithms that could potentially be distilled into lightweight elegance.
And if that ever happens AI really will be unstoppable, because we'll get instant red queen evolution that leaves us far behind.
The question is more whether alignment now can steer that towards less cataclysmic outcomes later.
All the more reason then to call their bluff. "Oooh we created a genie and it's almost out of the box". Cool, you've hyped the IPO, but also you have to plead your case before Congress as to why the company should continue to operate given its failure to prevent AI-related accidents from occurring.
Arguably the "social coding" angle is the reason why Tangled is the only actual alternative to GitHub. Otherwise there's actually plenty of other alternatives to GitHub, but none of them have GitHub's punchcard, which I'm ashamed to admit is the primary draw for me.
I'd like to know if most people use GitHub because of the social network part, or other reasons. I, for one, hate the social network part.
One thing with GitHub for open source projects is that the CI is free (and included macOS runners). The other is that many people have a GitHub account and would refuse to contribute to a project on another forge if it means that they must create a new account.
The federated design of Tangled solves the latter problem, so there's that.
I actually bought one of these and love it. My road bike is indistinguishable from any other normal road bike with the exception of it being much easier to get around now.
From experience I can say that it seems like a lot more common sense to simply replace the front wheel of your bike with a wheel containing an electric motor, rather than attaching whatever Asus is selling to your bike.
One theory I've been entertaining is that whenever GPT-3.5 came out a lot of people were talking about the "bitter lesson" and how scale was all we really needed to get to AGI. No need for any fancy tricks, just release a larger model trained on more data, by the time we released a hypothetical "GPT-5 sized" model we'd have AGI.
Anyway, the actual theory is that Google and Meta have fallen behind because they've been playing by this playbook of focusing on scale and training data, whereas OpenAI and Anthropic have done so well because they are likely doing much more interesting things to improve their models over time. It makes sense when you realize that one of Google's key strengths, besides talent, is that they have an incredible amount of data they can use for training due to being both the world's leading search engine as well as having all that video data from YouTube. Scaling the training data makes more sense to them than it does to Anthropic and OpenAI, who are both relatively data-disadvantaged.
You can kind of see this when you look at the Gemini 3 scorecard when it came out (https://blog.google/products-and-platforms/products/gemini/g...) and notice that while it wasn't as good as Claude And GPT at coding, it scored higher on a bunch of other non-coding benchmarks, and I think the reason why is simply because of Google's data advantage.
If true, I feel even more vindicated for believing that the "scale is all we need" narrative was bullshit.
Other than attention optimizations and other minor changes, the top Chinese models (which are way better than gemini) have basically the same architecture as GPT2. Of course RL is key for agentic workloads, but I'd say it's correct that progress has been mostly scaling models,adding more data and cleaning it better.
Google has a ton of smart researchers trying all sorts of stuff. They didn’t really go all in on any one thing. They hedge their bets.
But they suck at harnesses, developer tooling etc. their internal environment is very complex and an intern on a MacBook with codex can probably move faster than a seasoned deepmind engineer
> Anyway, the actual theory is that Google and Meta have fallen behind because they've been playing by this playbook of focusing on scale and training data, whereas OpenAI and Anthropic have done so well because they are likely doing much more interesting things to improve their models over time.
In case anyone has a fuzzy memory, back during peak NFT hype grifters would take art made by other artists and attempt to tokenize it as NFTs without the consent of the original artist, or otherwise possibly copy said artist's work and make a collection of it.
It was disgusting and parasitic then and it's disgusting and parasitic now.
reply