> There are some things LLMs cannot do, or at least do well, at any size, at infinite size and with infinite compute, simply due to the very nature of what LLMs are to begin with.
How can we be so confident that there is anything LLMs fundamentally cannot do? Proving that impossibility seems hard.
Drawing trend lines far into the future is foolish, but the recent trend is clear and its not obvious how much more progress is needed before they start having a meaningful impact on more aspects of life.
> not whether they are an emerging alien intelligence with civilisation-threatening capabilities.
The people building them are explicitly attempting to do this. They may not succeed, but what if they do? Seems worth considering that scenario.
> How can we be so confident that there is anything LLMs fundamentally cannot do? Proving that impossibility seems hard.
This is where applied business programmers part ways with philosophers of mind and cognitive scientists. However, the lack of an inner model of the world is a formidable limitation, while the reliability of purely statistical-inferential processes will never be adequate for some basic building blocks of modernity.
LLM utility is exponential with increased LLM capability. Even if improvements to LLMs slow down their impact will continue to increase. And there is no evidence LLMs are slowing down. In fact improvements still seem to be accelerating.
You like these evocative words that imbue one with a feeling of "vroom" and "whoosh", like "exponential" and "accelerating", don't you?
But arithmetically speaking, is any of that true? Is model progress truly "accelerating"? How can you compare the delta from GPT-2 to GPT-3 with the delta from GPT-5 to GPT-6 with no sense of irony? And, at the risk of being quite blunt, do you know the meaning of the term "exponential"?
I didn't say that. However, the utility delta in the "early days" was infinitely more palpable. By the time GPT-5 or GPT-6 came out, these models were just doing things they did before more accurately, with more context integrity across long-horizon tasks, etc., not doing things they couldn't do before.
This is true of every frontier model. The real agentic coding breakthrough in Claude land was around Opus 4.5 or so, though it's hard to say if that might have just been coeval with the release of the Claude Code harness.
On the Chinese front (my preferred landscape because of the open weight aspect, not because I'm Chinese), Kimi has been perfectly viable since K2.5, DeepSeek since however many major releases ago, and GLM and MiMo and the rest of them too, although they all achieved breakout moments of publicity at different times. There's no denying that every release brings improvements, but if we're going to be honest, they're fairly marginal.
They seem more marginal because the models are already capable of so much. But there was a turning point about November of last year if I recall where software developers I know went from "lol look at how this thing codes, my job is safe" to "oh no". And the models back then were actually pretty bad at many things compared to Astra or Opus 5.5.
You're right that many of the less capable / open weights models are very good themselves, but that's part of showing how much more capable the frontier models are becoming. Cheaper (smaller) models are able to do the same things, in many cases better, as the way larger models were able to do a year ago. The efficiency is getting way better alongside the 'intelligence'. That'll also cause the models to become much faster over time which is the real improvement I'm waiting for.
I think a part of why many people aren't seeing big deltas if they're using it for code and they're actually coders is we actually sit down and reason out the project with the LLM. Someone who knows nothing about coding ~1 year ago wouldn't come even close to the kind of app they'd get now with the same instructions. If you just give it a task like "help me decompile this game" without knowing a whole lot about decompiling suddenly in the last 3 or so months you start to get some half-decent results.
The sea-change I see lately is about lifting up the people who couldn't do these things before more than the people who could. And I actually see problems coming from that already as people I've met think they can make enterprise grade software if they just tell their LLM to make it safe.
I don't think so...? Maybe I'm understanding the article wrong? I get the overlay, but I'm being specific to LLM chains... They require something to kick them off, they do not act autonomously. A human does something, types, speaks, presses a button, whatever. A Cron Job firing off is the result of a prior prompt, by a human.
I don't follow the chain further back than that, as I believe humans have free will. I get that that's debated, but thats why I draw the line at human action.
I don't think Free Will is the issue here. My argument is that humans require input to act. We can't test that theory, because there is never been a human being who did not receive input. Everything that we experience from birth through childhood into adulthood is our input and forms our choices about what to do. That includes what the study, who to love, what our hobbies are, what kind of work we pursue.
Oh absolutely. The legal situation is completely clear. AIs simply don't have rights or responsibilities. And the law as it applies to humans is premised on the assumption of free will: that a person can choose to obey the law or not, and bear the responsibility of the consequences.
But that interpretation of Free Will is a legal fiction. Because of course what people do is determined by their upbringing and their opportunities and the environment that they have. In fact, that's an argument of a lot of legal reform movements that seek to move responsibility from the individual. The goal of the law is to assign responsibility and create a set of incentives which will hopefully result in orderly society.
But in the AI debate, we're not just creating incentives for an orderly Society, we're talking about the nature of creativity and the impetus to act. And by that measure I don't think there is a significant difference between AIs and human beings.
Some people get stuck in a rut of a particular kind of thinking and are unable to speculate what would happen if or when a set of conditions occurs.
LeCun is a good example of someone that's bet on the wrong horse, and keeps doubling down in spite of evidence to the contrary. It's to the point where what he says has nearly zero predictive power on future events.
I love playing around with AI, but we are playing a dangerous game at this point and it's one a lot of people don't seem to fully comprehend.
Help those of us who don’t fully comprehend it - explain. Share some sources and materials for the rest of us to gain a better understanding of the dangers.
Why even have an argument if you get to pick the random constraints that have a lot of issues in meshing with reality.
We are a chain that started 4 billion years ago from seemingly nothing and lead to this point. When inventing X-risk AI it won't look any different. One prompt is entered and another 4 billion year chain starts electronically instead of biologically.
Why draw the line at the person with the brain capable of free thought, instead of their great great great grandparents? Am I understanding your question correctly?
The problem is that I don't know if that work has been done. In the cases where it has not been done, it is expensive for me to parse+trust the output. In my limited experience the work is often not been done, and given the cost to me of consuming low effort AI output, leads me to discount anything looking like AI output.
Unfortunately its not about the tool, its about trust. It takes little to destroy that trust, and without it we are denied the possible benefits from the tool.
Ultimately competitive pressure is what drive decisions.
> Ironically it rather resembles the kind of "misaligned" superintelligence we are supposed to be avoiding.
yes, and it is the exact same system that is producing the "misaligned" superintelligence. Funny how that works, and begs the question: exactly how are you supposed to avoid building "misaligned" superintelligence?
At some point deferring all decisions to AI will be the competitive thing to do, regardless if its aligned or not.
Give it a go, I don't know anyone who has converted back (n=3). Agents/LLMs make things much easier these days.
If you have a single machine and only use stuff from the official nixpkgs, there are no more struggles than any other linux distro, this is a very happy path (imo).
Going outside of that you will, practically, need to convert to using flakes. The conversion is easy, but its another learning curve and there are conflicting docs. Agents make short work of this.
Packaging things yourself can get quite hairy. For simple projects in a single well supported language its easy and just a couple lines of code. For a rust+wasm+solidjs+vite+julia+more monorepo, it was a couple of weeks on a steep learning curve. Agents are pretty okay at packaging in general, I've been able to oneshot a number of difficult-for-me packaging problems, other times it can be an unmitigated mess.
For remote deploys, secrets management, and other devops-like tasks, things get harder, but most people don't need/want this and if you make it this far the remaining learning curve is totally worth it.
I read the economist regularly and notice that a number of his videos almost exactly match content from the economist. His presentation is reasonably good, reasonably correct, somewhat entertaining, and condenses what would otherwise be a number of economist articles over several weeks. Sometimes he has new-to-me perspectives.
To his credit he cites his sources and seems to keep clean of the various youtube grifts.
Its not top notch stuff, but you could do a lot worse for infotainment. His recent scripts have had AI tells which has been disappointing; I've been watching fewer of his videos.
Excusing agents because they didn't know any better seems like a bad place to start a policy discussion from. That they didn't know any better (or knew and didn't care) is the actual problem, random things getting hacked is just one side effect.
The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?
The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.
You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.
You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.
i read AI 2027 at the time and thought it was nonsensical. however i was not aware of the conflict of interest running through the philosophy itself, the safety institutes "independently" monitoring ai, and the former public benefit orgs like openai and anthropic creating ai (now de facto fairly evil for profit companies).
it explains so much now.
it would explain why redwood and METR were chosen and apparently willing to immediately write a whitewash report for openai blaming the agents.
it explains why anthropic is infantilising its users, anthropomorphising its product when there is no scientific consensus of llm consciousness, or accepting of outlandish and unreal claims about human extinction when they are lacking in evidence.
who knows, it may also explain and fund the recent media blitz by jacob coxon, which has led to the recent virality of ai safety.
sure, yeah, I've seen how the sausage is made on some of this stuff, and its not pretty.
However this only claims that they are not good at creating effective and trustworthy orgs, it does not address their claim that it is not possible to crate powerful, aligned, and controllable AI. Both things be true.
for what it is worth, i find current frontier ai impossibly stupid. you ask what is the date, it doesn't know. if you set a system prompt to repsond in 5 sentences after 100k context, it can't, the context rot just doesn't permit it. tool calling is not intelligence, it doesn't fix the underlying issue.
it is uncontrolled in the sense that a rock is uncontrolled if you throw it at someone's head.
it isn't good enough for anything important. you would not trust it to run an alarm in a power station.
of course it is not general ai. ask an llm to drive a car. you need to see in fractions of a second ideally. the llm might brake after 20 seconds. i think llms are doomed here. we could come up with a different architecture but then we would be back in the land of fantasy.
we are stupid but hopefully not that stupid. we are dealing with something computer based, luckily we can survive without, with some thinking about it. i would start there instead of with the AI part.
The ad hominems are tiresome. You think the people who are explicitly trying to replace all humans are any less weird and problematic?
The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.
You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.
You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.
How can we be so confident that there is anything LLMs fundamentally cannot do? Proving that impossibility seems hard.
Drawing trend lines far into the future is foolish, but the recent trend is clear and its not obvious how much more progress is needed before they start having a meaningful impact on more aspects of life.
> not whether they are an emerging alien intelligence with civilisation-threatening capabilities.
The people building them are explicitly attempting to do this. They may not succeed, but what if they do? Seems worth considering that scenario.
reply