Recursive Self-Improvement isn't instant, it starts slow and accelerates.
It starts with what they already claim to be doing - increasingly relying on existing models in non-trivial work related to training, evaluating and optimizing the next, more capable generation of models. As long as the proportion of work keeps shifting towards agents doing more and more of it, and humans less and less, that's RSI at play.
It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus, but frankly I find this notion absurd; LLMs already show better judgement than most people. The alternative is, at some point LLMs will show the ability to futz their way into improvement of the next generation of models even without humans in the loop - even if much less efficient at first, if generation N+1 is more capable than generation N, it'll either take off or burn out.
> It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus
All intelligence, LLM or not, is bound to plateau around the point where the need to operate within physical reality bottlenecks the speed of feedback. AI is progressing swiftly in the digital realm where feedback is nearly instantaneous, but it's unclear whether that would translate into improvements in the physical world where signals are much noisier and intelligence and judgment are less impactful.
Our tried-and-true approach for when the physical world is causing us trouble, is to remake the physical world to be easier to work with.
My go-to example: wheels suck at mobility in the natural world. There's a reason no animal uses wheels as their means of locomotion. They only really work on flat, hard surfaces that give a good grip.
Our solution? We didn't build mechanical legs for all-terrain mobility. We paved the world instead. Dirt roads, then brick roads, then asphalt and concrete - since ancient history, we were forcing the world to adapt, so the one means of mobility we could easily built was effective.
We repeat this pattern every time when the world doesn't agree with us, and adapting ourselves to it is too much of a hassle.
Now, AI will likely adopt the same approach to these kinds of problems, which may not end up very nicely for us.
(Note: we are already adapting our own digital worlds to make them easier for AI. Much like building roads for our wheels, we now have a resurgence of CLI, with new tools exposing high-level operations optimized for agents - not humans - to use comfortably.)
Sure, but is the limiting factor intelligence, here? Give human intelligence to dolphins and they're still not going to be able to pave the world. In order to remake the physical world, we needed certain physical abilities -- we needed the animal mind and body that took billions of years to evolve and embeds advanced priors about the world, and we needed it packaged properly. Irrespective of our intelligence, our body is a general world navigation device adapted to grab and move stuff. We use this innate tool to remake the physical world, but we didn't make it, and we'd be impotent without it.
With AI it's a bit of a chicken or the egg situation. Before it can ablate the complexity of the world, it needs to be able to deal with it, including adversaries such as ourselves. For this it needs a tool as sophisticated as our bodies, but could AI develop such a tool for itself without the help of this tool? I'm not convinced.
you are really just talking out of your ass here, no offense
just because more and more agents are doing human work, that in no way means the model somehow becomes magically more intelligent, it just means the work will stall and continue on at the same level forever
hell even if they hypothetically have an internal model that can output the entire training data set in a better format, there's no scientific evidence that the newer format has new information that is sufficient enough to train a better AI
as a matter of fact the scientific evidence is on the contrary
It's not the number of agents you should pay attention to, but the scope and nature of work they can perform effectively without being micromanaged by humans.
Even currently, there are clearly diminishing returns throughout the field. While not a skeptic myself, I think the age of LLM's is now over or close to plateauing. GPT-6.1 Sol seems almost the same as GPT-6 Sol(even if the benchmarks are slightly better). It doesn't help that we are running out of real training data, requiring new Synthetic data to be created for the next generation of Models. AI(or SI), has hit its peak, IMO, and now the focus shifts directly to Autonomous tasks. While Fable 5.1 is borderline okay at autonomous tasks, the amount of Slop it generates fundamentally requires a human to correct its mistakes. I was working on a website for a nonprofit with Fable, but the amount of slop and corrections I had to make was astounding/
It starts with what they already claim to be doing - increasingly relying on existing models in non-trivial work related to training, evaluating and optimizing the next, more capable generation of models. As long as the proportion of work keeps shifting towards agents doing more and more of it, and humans less and less, that's RSI at play.
It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus, but frankly I find this notion absurd; LLMs already show better judgement than most people. The alternative is, at some point LLMs will show the ability to futz their way into improvement of the next generation of models even without humans in the loop - even if much less efficient at first, if generation N+1 is more capable than generation N, it'll either take off or burn out.