There seems to be two common fallacies in SV technocrats' AI discourse:
(1) A far-reaching tendency to overextrapolate from the low-hanging fruit of the last few years of pretraining progress. GPT-2 to GPT-3 may have been a quantum leap, but GPT-3 to GPT-4 was not, and GPT-4 to 5 even less so.
The party has been kept going by RL and agents, but still, there is indeed a point of diminishing returns, not just relative to available compute but to how much training is possible when the entire intellectual output of humanity, plus a raft of synthetic data, has already been inhaled by the training process.
If one is to internalise the things that are said here on HN with regularity about model progress, and sentences ending with "yet" or "for now", then it would be easy to conclude that my 10 year-old son, who gained 3 inches of height last year, will be tallest structure on the planet by age 17.
(2) Inability to distinguish between technological, computational, and energetic limits of LLM capabilities vs. ontological / conceptual ones. There are some things LLMs cannot do, or at least do well, at any size, at infinite size and with infinite compute, simply due to the very nature of what LLMs are to begin with.
This latter topic receives almost no attention, except maybe from Gary Marcus and Yann LeCun. In that respect, this article is a breath of fresh air, insofar as it highlights that LLMs aren't "AI" at all, as we have traditionally understood the concept.
They really _are_ stochastic parrots. The relevant questions are about how much that matters for some domain or set of applications, not whether they are an emerging alien intelligence with civilisation-threatening capabilities.
> There are some things LLMs cannot do, or at least do well, at any size, at infinite size and with infinite compute, simply due to the very nature of what LLMs are to begin with.
How can we be so confident that there is anything LLMs fundamentally cannot do? Proving that impossibility seems hard.
Drawing trend lines far into the future is foolish, but the recent trend is clear and its not obvious how much more progress is needed before they start having a meaningful impact on more aspects of life.
> not whether they are an emerging alien intelligence with civilisation-threatening capabilities.
The people building them are explicitly attempting to do this. They may not succeed, but what if they do? Seems worth considering that scenario.
> How can we be so confident that there is anything LLMs fundamentally cannot do? Proving that impossibility seems hard.
This is where applied business programmers part ways with philosophers of mind and cognitive scientists. However, the lack of an inner model of the world is a formidable limitation, while the reliability of purely statistical-inferential processes will never be adequate for some basic building blocks of modernity.
LLM utility is exponential with increased LLM capability. Even if improvements to LLMs slow down their impact will continue to increase. And there is no evidence LLMs are slowing down. In fact improvements still seem to be accelerating.
You like these evocative words that imbue one with a feeling of "vroom" and "whoosh", like "exponential" and "accelerating", don't you?
But arithmetically speaking, is any of that true? Is model progress truly "accelerating"? How can you compare the delta from GPT-2 to GPT-3 with the delta from GPT-5 to GPT-6 with no sense of irony? And, at the risk of being quite blunt, do you know the meaning of the term "exponential"?
I didn't say that. However, the utility delta in the "early days" was infinitely more palpable. By the time GPT-5 or GPT-6 came out, these models were just doing things they did before more accurately, with more context integrity across long-horizon tasks, etc., not doing things they couldn't do before.
This is true of every frontier model. The real agentic coding breakthrough in Claude land was around Opus 4.5 or so, though it's hard to say if that might have just been coeval with the release of the Claude Code harness.
On the Chinese front (my preferred landscape because of the open weight aspect, not because I'm Chinese), Kimi has been perfectly viable since K2.5, DeepSeek since however many major releases ago, and GLM and MiMo and the rest of them too, although they all achieved breakout moments of publicity at different times. There's no denying that every release brings improvements, but if we're going to be honest, they're fairly marginal.
They seem more marginal because the models are already capable of so much. But there was a turning point about November of last year if I recall where software developers I know went from "lol look at how this thing codes, my job is safe" to "oh no". And the models back then were actually pretty bad at many things compared to Astra or Opus 5.5.
You're right that many of the less capable / open weights models are very good themselves, but that's part of showing how much more capable the frontier models are becoming. Cheaper (smaller) models are able to do the same things, in many cases better, as the way larger models were able to do a year ago. The efficiency is getting way better alongside the 'intelligence'. That'll also cause the models to become much faster over time which is the real improvement I'm waiting for.
I think a part of why many people aren't seeing big deltas if they're using it for code and they're actually coders is we actually sit down and reason out the project with the LLM. Someone who knows nothing about coding ~1 year ago wouldn't come even close to the kind of app they'd get now with the same instructions. If you just give it a task like "help me decompile this game" without knowing a whole lot about decompiling suddenly in the last 3 or so months you start to get some half-decent results.
The sea-change I see lately is about lifting up the people who couldn't do these things before more than the people who could. And I actually see problems coming from that already as people I've met think they can make enterprise grade software if they just tell their LLM to make it safe.
Necessary pedantry: if he's 4 foot 6 inches now, 3 inches is 5.555 (recurring) percent of that. If he grows at 105.555% per year then by 17 he is a fairly plausible 6 foot 6. (I think he then passes the forty foot mark soon after age 50.)
Ah, but he only grew about half an inch from 9 to 10, so the "rate" of "runaway progress" is "accelerating" "exponentially" (wait, how many derivatives is that?), and he will soon EsCapE CoNTaiNmEnT and hack HuggingFace...
I think a distinction is clearly emerging between docs intended for humans, and docs intended for agents. The latter is where all the Markdown-ridden slop should go; I'm not insulting humans with that.
I think so. Although it is possible that it is being held back for safety reasons. The same has been true of CPUs and many other advances, but the timeframes were generally longer.
I don't think any previous disruption was as destructive as this. Firstly it happens much faster and affects many more people at once, and secondly it doesn't seem to create any new jobs for the ones it replaces.
The industrial revolution took at least a century depending on how you count, and it created more jobs than it destroyed.
The introduction of computers obliterated whole office floors of employees at banks and insurance companies but it created even more work at companies that sold computers and software and consultant hours.
Ai takes millions of jobs and replaces them with a few thousand caretakers at megawatt datacenters. The workforce at Anthropic and OpenAI is comparable to a game development studio. If all they end up doing is reducing the workforce of JP Morgan by 10%, society still has a net loss of jobs.
That said, it's at least possible Ai will eventually end up creating a bunch of new work indirectly, by enabling people to do more than they ever imagined they could. One can hope.
Entirely agree with you, but I also lend some credence to the worry that capital owners and the managerial class might not want to pay a six-figure salary for "knowing the questions to ask".
I'd be lying if I said I hadn't spent any hours worrying about it. But on the other hand people pay for exactly that when it comes to law, accounting, actuaries, and whatever other white collar professions. I am nervous but optimistic for the time being.
They pay for tangible "deliverables" from those professions, too, not pure thought and judgment. One could probably argue that, as with programming, one has to have written a lot of code in order to think in code at a higher level, so to speak, although I can't say to what extent that's true in each of those domains.
However, the first part applies to programmers, too. A lot of the value of code _was_ simply in the fact that it was written, so you don't have to write it yourself. I'm not suggesting there isn't more to the value proposition, but the value destruction is there.
Fair, but the real question is what this person's manager thinks, as that tends to carry the day. And at the moment, there is an enormous gulf between what the managerial class believes LLMs can do and what people in the trenches know LLMs actually can and cannot do--not as a matter of current frontier capabilities, I might add, but more ontologically.
reply