But there's a new skill blossoming there, right? That's what I'm noticing. The atrophying skill is one you don't need anymore. The new one - prompting, routing the output, spotting the hallucinations, managing singal/noise etc. - is now where you can add tremendous value.
To say nothing of being bold enough to run systems like that in the first place. I know some of engineers that are still folding their arms and refusing to engage with AI!
SRE here, spotting the hallucinations requires you to have the knowledge to know when it's wrong. Best way to gain knowledge on systems is actually working on them. If AI do all the work, then you don't get the knowledge.
I don't know that that analogy fits for this comment. The argument is that the old skill is genuinely no longer needed in favour of the new skill, but the new skill requires experience from the old skill. I don't think being a good driver needs you to have already been a good cyclist.
I think it basically stems from the idea of “conceptual learning”. Which has been a pedagogical ideology for quite some time now. And is also convenient if you’re a bit addicted to your phone, a bit lazy due to AI, and want to still feel good about “learning” things.
We’ve known that one important component for learning (not the only thing that matters, but an important one) is the memorization of facts.
There seems to be a big urge to pretend that’s not true. E.g. you can learn arithmetic without memorizing your multiplication tables, literature without engaging with the canon, and now apparently coding without even having coded.
The idea is if we start with a top-down conceptual approach we will eventually vibe our way into knowledge acquisition. Which does sort of work.
However, sorry to be a curmudgeon, but the people who just “do the work” and actually spend time on task to learn a bunch of stuff … are actually more knowledgeable on average.
Good ones force themselves to do technical work hands on regularly, but will generally reach a level where they can't actually do that anymore unless they are really good at building management systems and delegating. For me it has happened in cycles over the last 25 years; after 4-6 years in management I refresh skills and go back to an IC role because I can't stand it anymore. I've done that cycle three times but I won't do it quite the same way again because its probably impossible to do without a pay cut now, and because IC don't write code anymore anyway.
This is both like forcing all IC to develop management skills but its also like the shift to cloud infrastructure or 4GL languages - we are never going back to the old ways and will be building everything at a different level of abstraction with a different span of control.
It's interesting how machines now mirror our relationship to sanity.
Just under the surface of human consciousness, we are constantly hallucinating and predicting what we may occur next. We are good at discarding what doesn't align with our working model of the world before reaching awareness and taking action. It's our attempt to conceptualize reality. Considering how un-grounded from reality humans are becoming, the fact we have to balance our nervous systems, the mistakes we do make day-to-day, and how minor physiological changes (from poison to injury to drugs to random chance) can disrupt the process, it doesn't take much for humans to make a major mistake or enter a psychotic state.
It wouldn't surprise me if a systematic approach to solve these problems with LLMs could be applied to human/hybrid teams.
I don't think that LLMs mirror human cognition or "sanity" as much as people want to believe. We are not constantly hallucinating in any way that applies to the term as commonly understood relative to LLMs or to humans. Mistakes, minor psychological changes and emotions aren't equivalent to LLM hallucinations or human psychosis. The relationship is superficial due to both humans and LLMs using natural language. But humans understand that language and its context, whereas LLMs do not.
The narrative exists to justify deskilling and replacing human expertise with LLMs on the premise that LLMs are at worse human equivalent and at best human superior but the hyperbole employed to make the false equivalence sometimes borders on the absurd and occult.
>It wouldn't surprise me if a systematic approach to solve these problems with LLMs could be applied to human/hybrid teams.
What you're describing aren't "problems" that need to be "solved" in human beings. They are problems when they persist in software, but we already had software that didn't have these problems.
This is a state of being that we chose because we decided profit superseded all other concerns.
Not even that Proton CEO claimed that. This is another one of those "no u" knee-jerk reactions littered all over this site, and I wonder when we'll first see stuff like "but humans also sometimes fail to load because of network issues", because there is no real distance between "humans hallucinate even more" and something like that.
A person has what is known as 'character' and 'personality'. These words indicate discenable patterns of behavior. Determinism in this discussion means that a given person will 'behave in an expected manner'. Further feature of humans in that you can coerce them to behave in highly constrained manners.
Entire institutions and organizations in humanity are based on this feature of human beings.
A heuristic software agent that will respond in a spectrum of possible ways and past performance is no guarantee of future performance. It is no way reassuring that your agent has 'never done x' since if 'x is in the space of possible responses' it can certainly do 'x'.
> Determinism in this discussion means that a given person will 'behave in an expected manner'.
Oh I see. That is not what determinism means. Determinism means their actions are predestined, "determined" by the initial conditions. Computer programs - including AI - are deterministic, barring cosmic ray flips. The same inputs running in the same environment produces the same output.
Humans are or are not, depending on whether free will exists.
Either way, it's hard to draw the boundary around determinism such that it includes humans but excludes AI.
No, determinism in this context clearly means precisely being 'predictable' as claimed that individual humans are predictable given sufficient observation.
I really don’t want to see code generated by Claude without some constraint like rules in a Claude.md file. Even Fable on high is lazy enough to forget a ton of things that a senior engineer would not. For example tracing and handling all unhappy paths, not repeating code, not writing terrible doc strings, etc.
We are using quite a big list of rules and the output is significantly more aligned with what we interpret as “good” code.
I don't know how to spot hallucinations, manage signal/noise, without having a pretty good understanding of the problem space. It's a tough balancing act.
No, everything you listed is low effort and minimal friction. It's exactly the luddite's greatest fear: to be delegated to a button pusher, to ensure that blinking green light continue to blink green. Anyone that can read and write can now add "tremendous value," so what do you think makes the value "tremendous"?
People cannot deny that cars have made our physical bodies atrophy. Not all of us, maybe, but many people are way worse off physically than they would have been without cars
There's an analogy in there but I can't think of it. Maybe I'll ask Claude if it can explain it
You seem to be overlooking the fact that Hegseth and other leaders in the US military and government have said that this is the end times or “armageddon”
true, but it's also super easy to switch models via Bedrock (my harness switches subagents between A\, OAI, Kimi, Grok, etc... GLM 5.3 is locked behind some dumb request system which I personally believe is because Amazon is heavily invested in A\) and you're not locked in with the Claude Enterprise features (i swear to god if someone sends another claude artifact instead of publishing to git...)
The singularity, as defined by Hinton (and others) as RSI (Recursive Self Improvement) may actually be beginning already, as OpenAI has announced an AI acting as a "research intern" (!).
How is this different from arguing that Microsoft Clippy was RSI? An AI tool being involved in the process of work can't be the bar for RSI.
I don't think there can be a coherent definition of RSI unless people lay out their theory for how intelligence scales. LLM-assisted coding is great but respectfully optimizing pytorch features or whatever is not gonna lead to exponential improvements. That approach to scaling diminished years ago, leading all the labs to switch to reasoning.
Now it seems reasoning is also yielding diminishing returns, so all the labs are pivoting to specializing in particular fields like math / infosec / biology. They're improving due to accessing new proprietary training data and doing RL with human experts. Again I don't really see any amount of "AI research interns" leading to an exponential improvement to this strategy, they're not the bottleneck in the first place.
>Now it seems reasoning is also yielding diminishing returns
Is the diminishing returns in the room with us?
>so all the labs are pivoting to specializing in particular fields like math / infosec / biology.
They're not pivoting to anything. The goal has always been creating a machine that could automate all or nearly all human work. They're just coming along on that mission.
As for RSI...I think the term is a bit odd in the modern context. It was created at a time when conventional wisdom was that generally intelligent machines would be these logic automatons that could "alter their own code". Instead we have massive neural networks that take months to train.
In this paradigm, the ways a LLM could "improve itself" would be altering its own weights directly or creating and training better, vastly more efficient architectures for the next generation of models.
The former is probably not happening but the latter is possible.
Yes, diminishing returns. Not overall, they've still been able to create more intelligent models even up to today. But the strategy for scaling that intelligence has shifted. From the initial ChatGPT release to GPT-4.1, they were basically scaling up compute training compute / model size. Then 4.5 flopped, while o1 demonstrated that gains could continue by reasoning (scaling up compute at inference time). o1 is now the ancestor of all their flagship models from GPT-5 on.
This is why I'm trying so hard to drill down on the theory of scaling, and not just talk about improvement in general, hand-wavy terms. If the bottleneck of current scaling strategies is training data, or something fundamental about the model architecture, then just throwing more harnessed chatbots at it won't lead to an exponential increase in performance.
Now you could argue that the AI we have now will help us find that change in architecture, and I would agree. But that means we're firmly outside the singularity for the time being, and what people are in fact talking about is a hypothetical.
>Then 4.5 flopped, while o1 demonstrated that gains could continue by reasoning (scaling up compute at inference time). o1 is now the ancestor of all their flagship models from GPT-5 on.
That's not quite right. They are still scaling model size and have had several new base pre-trains, just nothing so big as 4.5 (as far as we're aware). o1/4o has not been the base for some time now.
Data is obviously a bottleneck for some regimes and LLMs will have to get their hands dirty experimenting but it doesn't look like an insurmountable wall either.
There's no reason the reinforcement learning that is getting them better at computer use can't be applied to other domains, like biology, chemistry etc. It's just expensive, because the environment often becomes the physical world, it requires creating labs like anthropic are doing here, and gathering a lot of data, it requires llms attempting their own experiments(that's what 'getting their hands dirty' means).
Getting the data and setup will be expensive, but not impossible, and labs are clearly gearing up to do just that. If you can't understand that then that seems like a you problem.
> Now it seems reasoning is also yielding diminishing returns
Not true. On the contrary, LLMs are developing faster than predicted. They were expected to solve a Millennium Prize by 2030... and here we are in 2026. Release cycles are getting faster. Just compare the most recent GPT or Claude with what they were an year ago.
> How is this different from arguing that Microsoft Clippy was RSI?
We can argue about semantics, but that's not really the point. The point is that what started now - which no doubt is in its infancy - will result in full autonomy quite soon (they project an year or so), with the risk of RSI causing agent development to slip (long term) outside human cognitive control/capacity.
Again, can you lay out your theory for how intelligence scales? You're using a lot of terms like "full autonomy" without definitions. Why do you think that just throwing more harnessed LLMs at (something?) will lead to an increase rate of improvement?
I feel like I laid out several cases where other things were the limiting factor on improvement and more agents wouldn't have helped, and I didn't get a response to those cases.
What "they project" (the labs) is of minor interest to me. Aside from their incentives and track record of lying, in recent months they are laying out a story that is pretty much just the plot of Terminator, and directly referencing rationalist beliefs that were published long before LLMs even existed.
Self improvement during training, and AI self training are already happening. Easily/quickly are seemingly a factor of how much power/hardware you want to use at once.
With the level of compute they have they aren't stuck with frozen models like you are.
The infrastructure provisioning alone to train is heavily dependent on humans, as is dealing with failures (training runs fail a ton). Its not as simple as adding another ec2 on your dashboard. < 1k people in the world know how to do this, there will not be "recursive" or looped continual training for a long long long time. There are so many delicate inputs and controls. Not to mention the chains of businesses and the people required to operate them just to obtain the data needed, clean it and hand it to the llms.
The llms are supervising rlhf and creating synthetic data (to an extent) but they're nowhere close to being able to operate the full training stack end to end. This is a fantasy being sold to investors to create fomo.
Remember they're also limited by an effective memory of like 500k words a turn. Memory systems are lossy, so are swarm/sub agent mechanism. Im not worried about llms becoming self powered super entities anytime soon.
I think this is key. Lawyers make a lot of money for being experts on the sidelines of disputes with values greatly in excess of their fees. They were never paid for their busy work.
To say nothing of being bold enough to run systems like that in the first place. I know some of engineers that are still folding their arms and refusing to engage with AI!
reply