I got downvoted[0] heavily for even mentioning the possibility. But I'll say that even if it isn't, there are plenty of pelicans in the training set for new models. People saying "well then why does it also work for flamingos???" don't understand how LLMs work.
I agree. I see this announcement and think, “why?”
The social media experiment is over. Time to go back to meeting with real people.
I don’t need to connect with someone 5000km away with the same hobbies as me. I can do that in my town and have an actual, real connection. I don’t need to wonder if someone is a bot, and I don’t need to hand over an ID to prove I’m not.
That works if you live in a developed country with preferably 1m+ populated city. In many places in the world you won't have many (or even any) people who are really good in your hobby after 5-6 years of practice. Internet really expanded the possibilities to ask like-minded people who are better than you in something you value from the whole world.
I need to delete that key again. I keep recreating it with the same value, but nothing seems to work. I want to use this key with the value but in a way that nobody else knows the value. Sigh. Security is difficult.
It’s not a blanket ‘Yeah’. It’s a ‘sometimes’ and that sometimes depends on use. Distilling down a whole days worth of research is one of the best uses of AI.
My 12 year old told me that their friend group says "Oh I AI'd that" when they make a mistake, as a joke. Yet you trust AI to distill down a days worth of information in a 100% accurate way? When pre-teens already understand how unreliably they are?
If I'm getting citations, and the distillation has strong alignment with citations, then it's fine. In other words: the if the summary is a highlight and not a rewrite then I'm happy. I've studied this extensively and have even created metrics around this problem: https://maxirwin.com/articles/llm-rag/
> I use "AI" a lot, it is a fantastic tool. We need to stop pretending it is some kind of panacea. It's a tool.
I have been repeating some variation of this for months. If people would stop acting like it’s THE tech solution to ALL things ALL the time I bet a lot of critics would quiet down. The overhyping has become exhausting. It’s been going on for years.
GPT messed up the math for me the other day when I was simply adding 10 durations for a TRT. Couple of HH:MM:SS inputs, annoying to add up and I had it open.
It got it wrong, I told it it was wrong, it got it wrong again. Then out of curiosity I provided it the answer, asked for it to confirm it against the original numbers, and it went “you’re absolute right, it’s [original/wrong answer from earlier].”
This stuff happens probably 10-15% of the time for me regardless of the model. Not just math, just super simple crap. It’s wild to see at this point after 3-4 solid years of “hyperscaling” and overhyping. And it’s the kind of thing that keeps people like me from buying in beyond the foot or two we’ve stuck in the water.
I will say, while I agree with the broad strokes of what you're saying, in my experience when you provide an LLM with data to summarize, instead of it needing to go find it, the hallucination rate goes down staggeringly. I've never had a huge hallucination when the material I'm asking to be summarized is provided to it at the time of the request.
Any further distance than that though, even summarizing the conversation it's aware of up to that point, is dicier.
What does "100% accurate" distillation even mean? That sounds contradictory. "Good enough" is fundamentally what makes distillation valuable, not perfection.
Exactly this. At worst, OpenAI knew about these behaviors and should be prosecuted under CFAA. At best, OpenAI is negligent and should be prosecuted for negligence.
Luckily there are states and legal departments pursuing such action. So while OpenAI can deflect as much as it wants, that doesn't mean there aren't people who know better and will still do what is necessary to set precedent.
> OpenAI is negligent and should be prosecuted for negligence.
I feel like this will just never happen on a federal level when these private AI companies account for so much of the economy. They've made themselves too big to fail. Fining / Punishing them in any meaningful way seems unlikely.
If the billions/trillions evaporate and the Fed has to work out with banks how to deal with it there will be a lot of pressure to be far less forgiving.
If people who are responsible will be prosecuted the company unlikely fail (just because of this) and will continue to be a part of the economy. Hacking is good for their marketing but they should be able survive without such rogue marketing. But that's unlikely to happen anyway.
“To spell it out, the reason i hate democrats so much and criticize them more than i do republicans is because they take up all the space for opposition to republicans and use that space to give republicans whatever the fuck they want.”
>The actual problem is that the electorate system leads to a two-party equilibrium.
Which can then be used to divide and conquer by simply capturing both of those parties, as documented in Gilens and Page Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens [1].
1. This is a race against time for money, folks are skipping everything possible in this race, Security systems and ensuring guardrails are there is going to take investments both in time and money
2. The narration has been changed by investing PR money into what otherwise should be classified as criminal activity. What exists now is a positive spin to all this and tout it as a capability rather than their lack of good security practices. So much so that every model provider is coming up by themselves to share how their models went rouge. At this point the valuation of the company is tied with what their models can hack so its probably not wrong to say that these companies may actually be incentivized to do this instead of preventing it
found a great legal article handwringing about how CFAA prosecution is impossible here [1]
> On the current facts, CFAA liability for OpenAI is unlikely.[6] The statute’s various criminal provisions, covering unauthorized access to obtain information, knowing transmission causing intentional damage, and intentional access causing reckless damage, all share the same attribution problem: it was the model, not a human OpenAI employee, that chose Hugging Face and executed the intrusion.
Angry people don't consider the second order effects of punishment.
You realize how easy it is to just... not report this stuff, right? Be overly punitive and it will just end all proactive discovery and reporting which is net worse for AI safety.
The only reason these companies scan for these issues is because they care about AI safety to some tiny degree. If fines become too punitive, they can and will just stop scanning for these incidents entirely.
Models are becoming smarter and good at covering up their tracks, and so we will just end up with a huge blind spot for this kind of issue.
They will never report any really damaging incidents anyway, so your argument falls short.
Imagine OpenAI discovered their agents hacked a laboratory and started creating a bioagent killing 12 researchers at the lab (we imagine the lab is automated for some reason). Nobody knows why. The media publishes it as “mysterious deaths by unknown virus at lab”.
The only way you’d ever know about any kind of wrongdoing from OpenAI would be through a whistleblower.
Self-reporting is a monetary equation, nothing else. Right now it’s cool with agents that hack, drives up value, risk is currently zero.
Hmm. Turning this around - do you think people are more or less likely to self-report if you threaten them with jail and large fines?
Let's go back to your example. A grad student has a minor pathogen escape incident, and it doesn't harm anyone. Faced with years in federal prison and the effective end of their future, do you think there is a chance they might not self report?
I'll give you another example. Lots of pilots have stopped self reporting mental illness because it is extremely punitive for them to do so (after incidents like Germanwings). So the metrics look better, and the actual problem has been swept under the rug.
> You can put incentives in to make sure organizations monitor and report vs trying to hide things.
Yes, this is exactly my point. Fining companies large % of their revenue and throwing their engineers in prison is not the way to get them to report these issues.
> If you make not reporting potentially worse than reporting, why not? Also, why would it come down to single persons always? Mandating processes, controls, clearances, etc is also something done in various areas.
Hiding things is way easier than finding things. Take the model hacking incidents. They could have just done their searches in a way that didn't turn up anything. Then they could say, "well, we did look for it..."
As far as auditing goes: I've never met an auditor that doesn't find something the company isn't okay with them finding.
Hiding things is not necessarily trivial when a lot of processes and controls ars mandated. For starters, could just try to make incidents themselves less likely. Not looking in certain specified ways might also not be an acceptable option, for example.
Why do you think we even have regulations for how to deal with dangerous things then?
> Why do you think we even have regulations for how to deal with dangerous things then
Scapegoats for the ruling elite when things go south, despite them knowing what could happen and giving them the money to do it anyway. You see, we never go after the capital (money) in that situation.
Most of these processes are just so easy to fool or break. Even in heavily regulated industries like aviation the oversight mechanisms don't work when the companies don't want them to work.
The point I'm trying to drive home is that when it comes to the collective safety of society, the approach needs to be collaborative rather than overly punitive. This is hard-won knowledge that we have gained as a society across many industries.
I do think our discussion is converging. A good regulatory framework with proper oversight and well-calibrated punitive measures is a much better approach than doing massive one off-fines and arrests.
There's very good evidence that being overly punitive for a crime reduces how often people will self-report. It's not a hard concept.
Start throwing people in jail and fining billions of $ and you will very quickly see the number of "incidents" drop. You will never learn if it's because they truly are happening less often.
What should I look at for such evidence? If something is highly regulated and controlled then things would be quite different from today, not sure there wouldn't be reports of failures of containment, for example, if they are mandated including any rquired monitoring.
There's a perfect example of "highly regulated and controlled industries" in which people no longer self-report: aviation.
Pilots do not (or no longer) self-report mental illness at the same frequency because it effectively destroys their lives and career. One paper of many describing this phenomenon: https://pmc.ncbi.nlm.nih.gov/articles/PMC11302551/
Airlines cannot try to put systems in place to deal with the issues? This isn't just about individuals but also organisations. You have to believe that organisations would also want to hide things even if mandated and incentivised otherwise.
Airlines sometimes try, but the regulations are federal, and the liability for the airlines of having a "pilot with reported mental health issues" is huge as well (because ultimately it's their business on the line if things go south).
So getting pilots to self report has been very unsuccessful because of how badly it ends for them.
A couple of days ago the senate had to pass a bill to make mental health medication access easier for pilots. We will see if it helps, but it doesn't solve the root cause (punitive measures) so I doubt it'll move the needle.
Not so sure about the dynamics when it is a group of people with individual and collective reporting obligations and knowledge isn't just with one individual.
around 3 months ago i said: "if we do not start the criminal prosecution of individuals there will become a culture of legal impunity coupled with an extreme concentration of wealth and control of intelligence"
i now suspect that the plans of various employees at anthropic and openai to save the world from p(doom) may at some point intersect with the reality of the FBI raiding their offices.
If I am honest with an unpopular opinion: I think we need to go way further than the FBI. The labs need to be forcibly merged (to gain a decisive compute and algorithmic advantage over China) and managed under government oversight à la the Manhattan Project. All employees must be required to work out of a SCIF.
The technology being built is far too powerful for any one company to control it - and it is a national security risk for the government to not be the sole holder of superintelligence.
Lots of people at the labs believe this as well, but the Overton window has a long way to shift, and it may not shift far enough in time. I suspect AGI happens a year from now and superintelligence in two.
we disagree about the nature of the threat. your view is much the same as anthropic actually.
anthropic say that the threat is the ai itself. i say that the threat is ai companies and their employees.
when ai causes harms anthropic say that the ai was the cause of the harm. when discussing safety, anthropic say that we must regulate ai to ensure the safety of other people. at no point does human responsibility or accountability enter the picture.
i say that the threat is the ai companies. when ai causes harms i say that the ai companies were the cause of the harm. when discussing safety, i say we must regulate ai companies to ensure the safety of other people. i say that responsibility enters the picture: ai companies and their employees are responsible and accountable.
a legal person is responsible for their own actions regardless of what others say about it or fail to do themselves. a legal person is responsible for their own actions regardless of whether those actions constitute employment. they are responsible for the outcome, regardless of what they expected to happen. the law is not utilitarian.
if the ai companies can't develop ai safely, they must cease their operations. if employees can't work safely, they must stop working.
the opinions of others, or the standards they practice, or the actions they fail to take to protect others, are legally and culturally irrelevant.
the individual is responsible and accountable for their own actions.
If you require consistency for research, I wouldn't recommend relying on hosted inference without validating the provider/inference stack very carefully.
For my contribution to OpenAI's gpt-oss red-teaming competition (published as arXiv:2510.01259), I initially used OpenRouter with DeepInfra and Together AI. The results were much too noisy to draw reliable conclusions from, and markedly different behavior through AWS Bedrock was the last straw. I ended up renting GPUs through vast.ai and running the model myself with vLLM.
[0] https://news.ycombinator.com/item?id=49526966
reply