Agents are the greatest rubber ducks ever created.
Another perspective is that this isn’t a new problem. Every dev manager or team lead with very strong direct reports has this problem. They don’t lose the ability to reason, they strengthen their ability to reason more quickly, and know when to trust the expert and when to poke holes.
I don’t see how that’s possible. When I am doing a comprehensive code review with rigorous functional testing of another developer’s work, it usually takes me around half as long to review as the developer took to write it. That includes the back and forth of MR issues and fixes. If AI writes it why would it take 100x longer to review than write it? At worst case it’s just a draft of something I can develop myself, so it shouldn’t take longer than 20 minutes. At best case it is a code review exercise so it takes me half as long.
AI can produce for you an amazing jpeg of an oil painting you’ve ideated together. You could print it out on a large format printer but you’re still only holding a printout of an AI oil painting. To actually make a piece of art that’s worth being on a wall and standing as an artwork, then you still have to paint the painting using oil paint.
Maybe robot arms can solve this last part one day.
I’m not an artist I’m a software developer, so the tools I use don’t matter, the outcomes do. If I am completing my tickets and the code is high quality and well designed then I am doing my job.
Are you using the best models? I feel like my experience is completely different. I am an expert in a small part of a huge monolithic codebase that I’ve worked in for years. When customers report issues that would take me days or weeks to debug, AI can figure it out on the first try.
Yes, but it’s usually only hours to validate. I understand the codebase so I have the intuition to know almost immediately if the theory makes sense. Validating and reproducing the root cause is often an order of magnitude easier than identifying a production-reported issue.
>You ARE taking days or weeks to go and check, yes?
I've never taken weeks to go and check bugfixes in the before times, I don't see why I'd expect it now. Once we know what the cause of the bug is, validating the fix and writing a test for it is usually trivial.
I hate how the framing is always that “an AI agent” did something. No, an OpenAI researcher breached Medicare by using an AI agent. These tools only do what they are directed to do. The human operator is wholly responsible for what the tool does.
Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.
Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.
The biggest AI risk, by far, is the concentration of power in a few companies, located in the US.
There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.)
And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn.
The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.
I dunno, do you really want to bet that if you took a deliberately unaligned frontier model and asked it to permanently disable the US power grid, it would do a poor job of it?
If we focus too much on the world domination fiction, we really might, in a moment of lapsed attention, end up plugging essential infrastructure into the public internet, hoping some dream of global alignment might save us.
That's like giving a gun to a monkey and hoping the monkey is trained well. ..a frontier monkey though.
No. And no alignment, guardrails or slowdowns will ever secure our insecure infrastructure. Bad infrastructure decisions are a risk independent from AI.
The only way to secure infrastructure is to actually do the work needed to secure it. Our waste water treatment facility might not actually need to be able to tweet its status.
Cybersecurity is such an important topic for a country, dreaming about global alignment just to avoid fixing insecure infrastructure cannot be serious.
A Russian state backed hacker will not ask Dario for permission or argue with his LLM about ethics. That train departed long ago.
The nuclear arsenal is absolutely air-gapped from the public Internet today. And launching ICBMs is a key plot detail in many AI apocalypse fantasy scenarios.
Given that OpenAI and Anthropic have had major incidents in the last few weeks why will giving them a monopoly on models make thing safer?
The correct response is for these two companies to be closed down and their executives imprisoned. Yet somehow they have a free pass and are pointing at people who have not done any of the things they have and asking for them to be penalised.
Yeah, we should give a limited copyright waiver for pre training, given that the model is released as open weight and allow labs to compete with RL on that basis.
He did actually specify that diffusion should be limited for authoritarian countries, which I appreciated. If his focus is really safety, distillation in countries that are bound by safety regulation should be fine
I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation.
That section is so biased it reads like satire. The current US government is not safer or more responsible than China, nor is it a democracy. I can't tell for sure if this is a clumsy attempt to ingratiate Anthropic with the administration, or if he actually believes it, but given the "I agree with Secretary Bessent" name dropping I lean towards the former.
Another perspective is that this isn’t a new problem. Every dev manager or team lead with very strong direct reports has this problem. They don’t lose the ability to reason, they strengthen their ability to reason more quickly, and know when to trust the expert and when to poke holes.
reply