Extrapolating on the evidence that AI appears to have jagged intelligence, it is entirely possible (probable IMO) that ASI will plateau far above human capability in digital and symbolic intelligence, while staying far below in the ability to operate in the physical world.
As impressive as AI might be at virtual tasks like coding, virtual reality is remarkably different from physical reality in ways that artificially inflates AI results. One, it is far simpler. Two, the feedback loop is far, far quicker. It's possible AI could get the animal intelligence and animal body that lets us humans actually apply our intelligence in reality, but nothing in current technologies really indicates that this is a given.
AI doesn't need to be smart to seize our resources. They need brute strength, a proper physical intuition, and "hands". That may very well be several orders of magnitude harder than anything they're currently doing, we're just lucky it was in our starting build.
> it is entirely possible (probable IMO) that ASI will plateau far above human capability in digital and symbolic intelligence, while staying far below in the ability to operate in the physical world.
I don't think this is plausible, most notably because "AI research" is one of the tasks that clearly belongs to the digital world. You're suggesting that in the future all AI research is done by AIs because they are better at it than humans, and yet capabilities plateau at that point, instead of going into the RSI regime. (And it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics.)
The way I see it, if you made a large simulated world where virtual beings evolved and where each generation took a millisecond from birth to death, you would see extremely rapid evolution, but you wouldn't be particularly worried about these beings breaking out of the simulation and taking over the world, because fitness in the virtual world doesn't necessarily translate into fitness in the real world. If you create stuff within a toy world, you get toys.
I think the same is true of "research" and intelligence in general. There is no such thing as a general intelligence or a general capacity to research. It's always fine-tuned to an ambient setting, including ours. Even within humanity, our intelligence is most effective in fields where rules are rigid and quick and decisive feedback is available (math, engineering, computing), but it is much less effective in soft sciences, and disappointingly useless in politics. In other words, the dynamics of innovation and improvement are not purely mental, they are very much tied to external factors that are not always easily controlled.
> it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics
It's not "just physics", it's a honed system for surviving a realistic range of physical conditions, assess material properties on touch, adapt manipulation to arbitrary shapes, predict trajectories, take into account variations in ground, smell, wind, and so on; not to mention the machinery of the body itself, which self-repairs and protects us against a range of complex threats.
The point isn't that machines couldn't do all this. The point is rather that you can't quickly iterate and adapt designs to do it from an ivory tower. Pure "research" simply won't get you there, you also need to experiment, and insofar that physics limit the throughput of these experiments, the speed of improvement will be throttled severely. It is a mistake to think that "RSI" could break this, because it is categorically the wrong lever to pull.
I think you're massive underestimating intelligence - not ASI, but just, like, human-level intelligence. Most of the existence of human civilization has taken place in conditions wildly different from the ancestral environment, and we adapted anyway - and the reason why it was possible was because we have a tiny bit of general intelligence.
For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml - I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation, even given hundreds of years of theoretical research and several attempts in the real world? I think the closest real-world example of this is space exploration - it's extremely expensive and takes years to put stuff in space, and yet probes and rovers we launch end up working most of the time. On my model, it's because we have a decent grasp of the underlying physics in enough detail to predict real-world complexity, and we have toy testing areas we can test the tech on, and if you do this cautiously enough you can, in fact, launch a thing that ends up working.
I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
> human civilization has taken place in conditions wildly different from the ancestral environment
"Wildly different?" That's a wild exaggeration, if you don't mind me saying. Subjectively, sure, there are lot of important differences. Objectively, they're basically the same. The intellectual environment machines bask in is many orders of magnitude more different from physical reality than living on Earth is from living on Jupiter. Even the topology of it is without common measure.
> and the reason why it was possible was because we have a tiny bit of general intelligence
Also, hands. Also, an upright posture. Do you think just any being with the level of intelligence we have would have followed the same path? Attributing our success solely to our intelligence is simplistic -- it is our intelligence plus a slew of innate capabilities, which we did not create, and without which our intelligence would be worthless.
The kinds of simulations you find in LessWrong's half-baked thought experiments are almost always ridiculous and this is no exception. "World simulations" in real life are massively limited stuff like Minecraft where you might be able to embed a 1 MHz computer if you manage to bump it to run 20K ticks per second. The resources required for the simulated beings to do all that work wouldn't be available and their generations will be slower than ours, not faster.
I didn't spell it out, but the simulation I had in mind was for a toy world, because only a toy world could possibly run fast enough.
> I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation
I'm talking about operational, practical, instinctual knowledge for real-time action. Knowing physics doesn't make you capable of catching a ball, and you don't need to know physics to catch a ball. In real life, we never deal with the fundamentals of physics, we deal with macroscopic objects and systems that are very far removed from these fundamentals. They are more varied, messier, faster, and overall harder to deal with.
> I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
Blind optimization has one definite advantage over careful planning, however: it is not self-limiting. Any goal-directed optimization process is de facto limited to the subset of possibilities that it can prove furthers its goals. That's why most human technology is simple and unsophisticated compared to natural systems: it needs to be predictable, reliable, build-upon-able, and therefore easy to model.
And when it is more sophisticated, like current AI, it is so heavily inspired from natural optimization processes that I would argue it is actually a form of blind optimization. I mean, think about it: we understand AI so poorly that you and a lot of other people are afraid of losing control of it. If we were designing it intelligently, we wouldn't be worried. But we aren't: we are doing it blind.
The best problems to work on are not necessarily the hardest ones, nor the ones that need the most intelligence. They're the problems you, or other people, actually have. Are you going to give up on painting your deck because it's too easy and you don't need a 500x genius to do it?
But if you apply the logic that led to the conclusion of "natural equilibrium", truth of the matter is that we wouldn't be able to say if the waiting will be over before or after the heat death of the universe.
Even the "million years" is an overoptimistic view, and probably reflects the idea that the ecosystem was in some kind of messianic state of equilibrium before man arrived, which happened roughly in the last million years. But even that is broadly a fiction. Ecology is not a system, and it is not stable even without human intervention.
Yeah, it's a bit absurd on its face to speak of equilibrium in the context of evolution, which is basically a litany of innovations and equilibrium breaks. Nature regularly throws itself out of whack, like when cyanobacteria appeared I recall, and then when nature made us, obviously. Whatever equilibriums existed through time are rarely returned to once broken.
I increasingly feel like "consciousness" is little more than "that thing we humans have... you know what I mean... you have it too!" and that any attempt to elevate it to a more general principle is doomed to failure. Sure, we can easily extend it to other living beings because we share a rich evolutionary history, but beyond that...
As far as AI goes, I think that whatever it is that they have is analogous in significant ways, but also irreconcilably different in significant ways. They aren't proceeding in a straight line towards consciousness either, because why would they? They're doing their own thing which happens to produce similar outputs. It's a problem in terms of value: we value consciousness, because we value ourselves, because we value our output. If it turns out consciousness as we know it is just a minor implementation detail with no cosmic or long term significance, then irrespective of whether AI is conscious or not, we still have to grasp with the challenge of explaining why consciousness is valuable in the first place if not for its outputs.
I'd argue that the reason why we value life, consciousness, the ability to suffer, was always arbitrary, and it relied a lot on the unicity of human experience. It was never robust to the breaking down of human capabilities into inspectable, reusable, upgradable and resettable modules. Or the radical decoupling of capabilities we hitherto considered correlated, like writing about love and feeling love, and many other things.
I only read the first book of the trilogy and the "science" in it was so profoundly absurd that I couldn't motivate myself to read any further. I think part of the issue is that it had been sold to me under the guise that it was hard sci-fi, and it was anything but.
I disagree it's unoriginal, though. It's pretty original, you just have to be prepared to suspend your disbelief (it's kind of on me that I wasn't when I read it).
Is it? Maybe I didn't get far enough into the series, but the parts I did read and the synopsis online are all derivative of old ideas. The "dark forest" idea is a very old and much debated solution to the Fermi paradox, and appears in other works (e.g. the wolves in Alastair Reynolds' Revelation Space series). The VR stuff was derivative of dozens of previous works. The rehydration of the solarians is not that different than the spiders freezing in A Deepness in the Sky by Vinge. Plenty of alien invasion and doom cult literature out there (too much, in my opinion).
But I took Dark Forest to mean the Dark Forest Theory which the post I was replying to mentioned, and that's just a common theory under a new name, which has been a commonly known solution to the Fermi paradox for so long that no one is credited with coming up with it. It's given the "deadly probe" name in a SETI review of the Fermi paradox literature in 1984.
My criteria for originality may be more lenient than yours. I think it would be difficult to find sci-fi that isn't derivative of older ideas to an extent, and it's harder for more recent work to stand out, because basically everything's already been done. Managing to combine a dozen ideas from previous works into a coherent whole is far from uncreative. So I feel the author was original enough in his approach, even though I thought it was scientifically inept.
I think any alternative system has to figure out how to attract and reward the kind of people who are intrinsically motivated by truth, discovery and knowledge, and repel the kind of people who aren't.
The chief issue with external incentives like "fame" or "money" is that they will attract a lot of people who care about the reward, but not the work. If they can find a way to cut corners, they will. So it works fine if you have a very tight, objective and reproducible evaluation criterion such that you can't get rewarded for poor or sloppy work, but even in science this is difficult to establish, especially in a crowded field where most advancements will be small and incremental. Science works a lot on trust, if only trust that the described experiments were indeed made and the results were indeed as reported.
That's why I feel we get the best science when it's driven by people who just intrinsically want the best science, because getting it right is a reward in and of itself. It'd be neat if it was possible to identify them and just let them cook, regardless of whether results are produced or not. Paradoxically, I think this may be easiest in a setting where academia is undervalued -- the knowledge-seekers will always self-select into research (provided the conditions are good enough and they are not too undervalued), but the fame-chasers will have to find their validation elsewhere.
> Already businesses that have more compute and access to data seem to eat the world around them.
Do they? Business valuations are more a reflection of what people believe will yield returns than a reflection of reality. A lot of big tech behemoths could disappear from the face of the Earth overnight and it would make little difference to our lives.
Natural resources and energy are the actual backbone and it is a true catastrophe when these are disrupted. In comparison, disruption to compute or data is just an inconvenience.
> It may be that it turns out LLMs lack some fundamental level of judgement and it plateaus
All intelligence, LLM or not, is bound to plateau around the point where the need to operate within physical reality bottlenecks the speed of feedback. AI is progressing swiftly in the digital realm where feedback is nearly instantaneous, but it's unclear whether that would translate into improvements in the physical world where signals are much noisier and intelligence and judgment are less impactful.
Our tried-and-true approach for when the physical world is causing us trouble, is to remake the physical world to be easier to work with.
My go-to example: wheels suck at mobility in the natural world. There's a reason no animal uses wheels as their means of locomotion. They only really work on flat, hard surfaces that give a good grip.
Our solution? We didn't build mechanical legs for all-terrain mobility. We paved the world instead. Dirt roads, then brick roads, then asphalt and concrete - since ancient history, we were forcing the world to adapt, so the one means of mobility we could easily built was effective.
We repeat this pattern every time when the world doesn't agree with us, and adapting ourselves to it is too much of a hassle.
Now, AI will likely adopt the same approach to these kinds of problems, which may not end up very nicely for us.
(Note: we are already adapting our own digital worlds to make them easier for AI. Much like building roads for our wheels, we now have a resurgence of CLI, with new tools exposing high-level operations optimized for agents - not humans - to use comfortably.)
Sure, but is the limiting factor intelligence, here? Give human intelligence to dolphins and they're still not going to be able to pave the world. In order to remake the physical world, we needed certain physical abilities -- we needed the animal mind and body that took billions of years to evolve and embeds advanced priors about the world, and we needed it packaged properly. Irrespective of our intelligence, our body is a general world navigation device adapted to grab and move stuff. We use this innate tool to remake the physical world, but we didn't make it, and we'd be impotent without it.
With AI it's a bit of a chicken or the egg situation. Before it can ablate the complexity of the world, it needs to be able to deal with it, including adversaries such as ourselves. For this it needs a tool as sophisticated as our bodies, but could AI develop such a tool for itself without the help of this tool? I'm not convinced.
> What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
For what it's worth, ten thousand terawatt fusion plants probably approaches the level at which the sheer intensity of energy production would cause significant disruption to the climate (it's roughly 5% of the Earth's entire solar input). Every energy source becomes dirty past a certain point. It would be wiser to learn how to build a utopia within a limited energy budget than find a way to produce enough of it to cook the damn planet, but who am I kidding, we're going to build a million of these things.
If we had such arbitrarily large amounts of energy to handle, it would be conceivable to beam excess energy into space. It is a bit on the sci-fi side, but so is that kind of energy production.
This assumes other technological breakthroughs don't also happen in parallel. I'll go out on a limb with nothing more than a hunch that we are in the beginning stages of understanding mathematics, physics, and biology.
It does not. This is basic thermodynamics. Unless we are using computers made out of something other than matter, powered by something other than energy, thermodynamics bounds us all.
Ok, but what if the AI figures out a new thermodynamics? A completely new thermodynamics, that's vastly more powerful than human thermodynamics, but of course humans won't be able to comprehend it, because it's so much more powerful, that you need the superintelligence to handle it. With a thermodynamics like that, anything could be possible, and remember: what we have now is the worst it will ever be.
Even today we're recognizing that compute does not need to happen here.
When we're talking about creating powerplants equal to roughly 5% of the insolation of Earth, I think we're sci-fi enough to discuss orbital datacenters or Mars datacenters or Jupiter fusion candle datacenters.
Orbital datacenters suffer from the same problem: limited surface area. You think cooling is a problem now? Try not having conduction or convection. Radiation is all you get. A huge proportion of the mass and size of the ISS is radiators, and that only has to deal with the heat of low-power electronics and a few measly human beings.
Using an entire Mars only nets us about a 2x multiplier for our energy expenditure budget. Then we need four planets to double it again. Exponential growth is a bitch.
Thermodynamics and the tyranny of exponential growth are going to win this battle every time, regardless of the unobtanium technology you try to invent.
Not quite the same problem, because only a small portion of that energy radiated from space will make it back to the environment on Earth. For convection that is 100%.
> Using an entire Mars only nets us about a 2x multiplier for our energy expenditure budget.
That is not the problem being discussed. Radiators can be scaled up, but the thermal mass of Earth is finite. GPUs can survive at temperatures the organisms of Earth cannot.
I am quite certain that you are drastically underestimating the scope of the problem of "getting rid of heat" when you're dealing with the discussed levels of power production.
Maybe, but that's a "simple" resource problem. The Earth radiates 20x more heat than that today, and it has not been optimized for purpose. Remember that the initial assumptions were that AI has already designed and manufactured tens of thousands of these plants.
The problem at hand is not just getting rid of heat however, it's getting rid of heat without cooking the Earth. I still fail to see how generating the heat off planet fails to solve that.
Compute doesn't need to happen here, but it can, so it will. Building datacenters on Mars or Jupiter doesn't reduce our ability to build datacenters here, so naturally, they will still be built, everywhere that they can be. It's an orthogonal capability.
Maybe, but there are counterexamples today. The processing of raw materials can be done all over the world, but it's not. Modern nations actually disassemble their blast furnaces and ship them out at a certain point.
It's quite possible that with "infinite" energy we could do mass scale carbon capture to offset the increased temperatures by rolling back the greenhouse effect.
This is basic thermodynamics, not greenhouse gases. If you put some amount of energy into a system with a fixed surface area, you will get a minimum equilibrium temperature.
Greenhouse gases and Earth’s internal nuclear decay engine make the situation worse, but even without them this would boil the oceans.
The more I think about space elevator, the less I believe in it.
The material alone is in a quantity beyond what we can reasonably manufacture.
and the material needs to be perfect. All design we have today have cascade failure mode -- any material failure translates to a total catastrophic failure.
and geostationary does not really meant Geostationary. There are lots of jiggling everywhere. It wear down over time. and let's hope nothing resonance
and we need some maintenance / decommission plan. How can we decommission this when it fail or need upgrade?
We're not going stop doing something locally just because we can do it in space. There are definite physical advantages to producing next to consumption. We will most likely do both -- things that can be done in space will be done in space, and things that are better done locally will ramp up. There is no upper limit to energy demand, none whatsoever. Stopping production anywhere it could theoretically happen is a matter of will.
I don't think "abundance" is a stable state. There are always ways to use more resources and more energy, so no matter how much we have, it will never be "enough" -- abundance is just a transition state where a population or system has not yet figured out how to grind the surplus. The faster the progress, the shorter I would expect these periods to be.
As impressive as AI might be at virtual tasks like coding, virtual reality is remarkably different from physical reality in ways that artificially inflates AI results. One, it is far simpler. Two, the feedback loop is far, far quicker. It's possible AI could get the animal intelligence and animal body that lets us humans actually apply our intelligence in reality, but nothing in current technologies really indicates that this is a given.
AI doesn't need to be smart to seize our resources. They need brute strength, a proper physical intuition, and "hands". That may very well be several orders of magnitude harder than anything they're currently doing, we're just lucky it was in our starting build.
reply