I keep seeing this argument. But isn’t the inverse true as well - if you extrapolate from the past movement of the ball, you will see it going way farther than it will actually go because of friction, wind, someone catching it, etc.? It seems reasonable to look at where the tech is right now if it helps you understand fundamental constraints. A person witnessing the first flight as a child and the moon landing in adulthood might expect FTL travel for their children if they didn’t understand physics.
Both my grandparents raised 3 kids on a single salary and owned their own homes by thirty. One of them did this in a communist country!
They never had laptops so the work finished at 5 and that was it. Ditto for weekends. I work longer hours and could never dream of doing what they did with two salaries.
The book Capital in the 21st Century shows that the phenomenon you're talking about with your grandparents is a historical aberration caused by World War 2 and is unlikely to be repeated again unless there's another destructive war.
Once you hit rock bottom the only way to go is up, hence creation of wealth for many Americans and destruction of it for many Europeans. Policy choices don't exist in a vacuum and are instead beholden to the conditions of the world, one of which being, it's very easy to build up when everything around you is flattened.
And before that? As I said, your grandparents are an anomaly. You are regressing to the historical mean. Policy can control some of it but black swan events like world wars do occur.
I’m seeing this perspective expressed a lot lately, but (so far) only online.
Understand that there’s more than one way to prove something, that there’s an art to making something understandable, and that humans tend to take the most understandable paths (to humans.)
Models often don’t do that. But in solving, or claiming to solve, important problems, they nevertheless remove a lot of the excitement and incentive towards working towards a solution humans would naturally find. Plus, a lot of these problems really don’t have any critical applications at the moment. They’re done for the love of the game.
Does that bother you? That your tax dollars pay people who do math out of curiosity?
Most feel-good pieces on AI make the same mistake, which is looking at where the ball is right now and not where it's going. If progress were to halt today, this article would be relevant and I'd upvote it.
But since the richest and least-trustworthy people on earth are on track to make an ultraintelligent thinking machine, the risks I'm worried about are: killing most jobs, concentrating most wealth, and putting ourselves at risk of literal extinction. All I could think when reading this was, "this person really did not think this all the way through." I'm sorry.
> Most feel-good pieces on AI make the same mistake, which is looking at where the ball is and not where it's going.
Humans have a bad track record predicting where a technology is going.
I do not know where AI heads in the next 5-25 years (I have my hypotheses, but they are likely off, too), but I see a useful tool that allows many people to experiment with and try things that they could not before and I do not see why should I subscribe to the doomer view.
As far as the moral character of the leaders: that's common, too. Many luminaries were, based on recollections, seen as pretty bad human beings and that didn't run off on their inventions. My 2c.
Being a doomer sucks, man. But I see a lot of issues with your way of thinking, which I was basically getting at in the post you replied to. Let me explain:
Today it's a useful tool. But in not too long--a few years, perhaps--this tool will be able to outsmart you. If you think this is definitely not going to happen, then okay--we see the progress in LLMs differently. I see machines which, in a matter of three years, went from being able to do basic proofs, to math Olympiad problems, to century-old Millennium problems. I see no reason why progress will slow down.
Any "tool" which is that much smarter than you is no tool, any more than you are the "tool" of your pet dog.
I understand wanting to feel better and be optimistic about the future. I'm unfortunately not, and that makes a lot of people feel bad. Again, I'm sorry! I don't want to be a downer. But it's important to me that people take these risks seriously and don't minimize them with expressions like "we still have paper" and "it's a useful tool."
> But I see a lot of issues with your way of thinking, which I was basically getting at in the post you replied to.
Of course. And I am not claiming that my view is the view or even a particularly good view. It is just my personal worldview. But I think that with ups and downs, over generations and centuries, the world is on an uptrend. My parents were being driven, with their families, from their homes as kids; one of them lost an eye (as a kid) from a war; both lived through years of hunger. My grandparents saw real crap, two lived through a famine and saw many die from hunger.
But they all lived through those disruptions, worked and tried to build something better (I had to probe that information out of them; they did not even mention those things to us). I think the AI disruptions will not be nearly as painful and we will live through them just fine. Again, just a personal perspective.
> I see machines which, in a matter of three years, went from being able to do basic proofs, to math Olympiad problems, to century-old Millennium problems. I see no reason why progress will slow down.
I think what we see is LLMs picking the low hanging fruit. There is a lot of it, but I do not think we can extrapolate it. Some things will be particularly easy for LLMs to exceed humans on (like a steam engine being better in driving heavy machinery), some possible, many very hard. And humans will find new things to explore and build. We will see in 25 years (I am old and might not be alive then, but maybe I will see enough in 15).
As a side note: I do not know why you are being downvoted. I do not agree with your view, but I think it is intelligent and well written. I hope folks stop downvoting anything they disagree with and go back to downvoting on tone, not alignment.
>But I think that with ups and downs, over generations and centuries, the world is on an uptrend.
I mean if you were a dinosaur you had a 100 million year uptrend and then it was suddenly over.
Things did not becomes better on accident, in general humans have wanted a better world for themselves. There is no scientific rule that says this must remain true. We have to strive for it. And if we hand over all this to a few rich people with intelligent machines there could be a huge upset here.
> I mean if you were a dinosaur you had a 100 million year uptrend and then it was suddenly over.
Yes. But my point is that we have a major crisis (and a part of the population thinking that the current way of life is over forever), every few generations. With your analogy, the dinosaurs had "the end is nigh" and "this time is different" doomerist beliefs a million times before finally being right.
Agree fully that it is good to strive for the improvement. But there are so many different views of what the improvement would look like, so we have good folks who, when spun up by their respective leaders, focus on their differences and start fighting. Which, IMO, is almost, almost, almost always a bad way to build something better. My 2c.
Calling out doomerism as some kind of unrealistic disaster thinking is a pretty shallow understanding of the modern world and how much anti-fragility as been removed from different societies.
The world from the 90s till now has been one of rather open trade and exchange of energies like fuel. In that time the population has grown by at least 3 billion people. Billions of us depend on fuel being available every day or a countdown until there is no water or food starts. That fuel system is dependent on power being available to run things like pumps.
>doomerist beliefs a million times before finally being right.
Because you get zero chances of being wrong. Eventually humanities time will be up. I would rather that be millions of years from now rather than a few years because we collapsed under our own hubris.
All I hear when I read the “but it’s just a useful tool” takes is “hey it’s fine if we use LLMs to kill people and take away their dignity and ability to make a living because it generates documents for me”.
Wealth is already concentrated. They’re just coming for the top 10% that affect the software devs now since they’ve taken everything else. You talked to anyone with an “unskilled” job in the last few years?
The thing that strikes me about the 'paperless office' as it was hyped isn't how much paper use was reduced, but what was enabled or sped up by computer databases or the internet to work with a head office. Consider how restricted most mail order companies would be compared to the modern market if they could only communicate with their customers, suppliers, logistics and other partners via phone/fax/physical mail, and they either had one central records department or the difference in efficiencies handling a network of local branches/franchises.
That leads to pondering if we're actually seeing a similar shift today with AI.
> They're going to do all those things with or without AI.
Yes, but AI gets them closer to those things faster.
> We need to fight the end goals at a social level rather than the temporary means
Again, yes, we need to fight the end goals, but we also have to stop the harm caused by the means. If there’s an arsonist going around burning people’s homes we do have to catch them but we also need to put out the fire on the houses burning right now (imperfect analogy, sure, as they all are).
> are on track to make an ultraintelligent thinking machine, the risks I'm worried about are: killing most jobs, concentrating most wealth, and putting ourselves at risk of literal extinction. All I could think when reading this was, "this person really did not think this all the way through." I'm sorry.
While two out of those three risks do not require anything "ultraintelligent", for killing most jobs it's worth remembering that their track is an appropriation of intelligence, and the speedups happen where it's easiest to measure more than where the tracks correspond to economic value.
Unfortunately, this has the opposite effect for safety, and can still do a lot of wealth concentration even if it doesn't end all jobs.
I can think of a dozen jobs off the top of my head that aren’t even remotely at risk due to LLM-style AI. And if your reply is, it’s only a matter of time until humanoid robots replace all human jobs, you’re very divorced from reality. In hundreds of years, maybe. Not in your lifetime.
This tech will already have a massive effect on society, purely from being able to generate text and software code at will. What is the constant need to exaggerate this into a Sci-fi story?
Also if that many people were put out of a job, to the level that their basic needs were no longer met, there would be mass unrest/revolution and all of the robot production and data centers would be burned to the ground. See: French Revolution, Russian Revolution, etc.
And it would almost certainly happen before it got to the point of "they have enough robots to stop the masses". And probably we'd hit 3.5% rule[1] non-violent government regulation of AI before any of that.
The real risk of AI to me is that we're too heavily invested in it and when the bubble pops it could take a lot of institutions with it. And governments are not in a great position to bail economies out again.
You live in a sci-fi world, we are that sci-fi story. Nothing we're talking about is outside the world of scientific possibility.
One thing you are really missing is a small amount of job displacement is going to create a huge deflationary amount in wages. At the end of the day you don't need that many plumbers and electricians. There was a reason a lot of people moved to office and tech jobs starting in the 80s. Trades were a dead end at the time. And the same thing will happen again as labor shifts.
This is making the same mistake: looking at where AI is now and not at the rate of progress it’s made. If you have a thinking machine that is more capable than every human on Earth and thinks 100x faster, how long do you think it will take for it to design humanoid robots.
I don’t think designing is very difficult. It’s actually having the resources and economic capacity to make millions of them, enough to “replace all jobs.”
It’s also a fallacy to assume that current progress implies future progress at the same rate. This is very basic technological history.
Again, a sci-fi fantasy for at least the next century.
> It’s actually having the resources and economic capacity to make millions of them, enough to “replace all jobs.”
Like that newfangled invention called the automobile. We could never produce millions of them to substantially replace all other means of transportation.
EDIT: Joking aside, I think that when humanoid robots get good enough (I'm predicting 5-10 years), there will an insatiable demand for them in every possible residential and commercial application. If we have as many cars and portable supercomputers (i.e., cellphones) as we have people, why is it hard to imagine producing as many robots as we have people?
Extinction will come not with a bang but with a whimper. The boundary of the burgeoning poor — already at two nines — will expand to include pretty much everyone, and that underclass will dominate society to the detriment of everything in the commons: culture, civility, affordable food, education, thinking.
Western culture devolves into poor, corrupt, violent, petty criminality. Welcome to Mega Cairo.
We're not living in the same universe. Look at capability year-over-year. The asymptotic maximum is wishful thinking. Someone needs to step in before this gets wildly out of control.
Year-over-year, yes I would agree with you, but that is not the cycle time of models. And while I share the same hunch as GP, I'm waiting another cycle before putting more confidence behind it.
> Someone needs to step in before this gets wildly out of control.
This sounds like a paid bot, lmao.
I wonder how much is actual improvements in the model and how much comes from better harnesses. I'd be willing to bet that the combination of better harnesses + the models being trained around agentic workflows (i.e the harnesses) has made it significantly better in many things, but it's hard to find other improvements imo.
You can see the difference between GPT 3, 4, 5, 6 each year. Same with Gemini and Claude. Look at image and text generation. Code generation. Math ability (Navier Stokes, Quasi-Riemann.)
I don't understand the point of this debate. Rich's point is good. Having a mini-army of highly intelligent beings at your fingertips is not an ordinary "tool."
And it minimizes the fact that these things are only getting smarter. Having an entity much smarter than you--could that entity seriously be considered your "tool"? Just your tool? Or are the people on the "just a tool" side of this debate not taking that possibility seriously?
It might be better to view the near-future AI as an emerging intelligent alien species.
Regardless of whether it is 'conscious' or not, Roman Yampolski makes the point that AI may or may not have human-kinds best interests at heart [ hell, even humans aren't aligned ]
Can we control or trust a counter-party that has intelligence at a level near or exceeding the smartest humans in all domains ?
I dont even think we can even have a safe sandbox in which to run an AI .. and frontier model training labs are certainly not air-gapped.
In the 1960s there was recognition of some of the implications of
human intelligence. I. J. Good wrote [10]:
Let an ultraintelligent machine be defined as a machine
that can far surpass all the intellectual activities of any
any man however clever. Since the design of machines is one of
these intellectual activities, an ultraintelligent machine could
design even better machines; there would then unquestionably
be an "intelligence explosion," and the intelligence of man
would be left far behind. Thus the first ultraintelligent
machine is the _last_ invention that man need ever make,
provided that the machine is docile enough to tell us how to
keep it under control.
...
It is more probable than not that, within the twentieth century,
an ultraintelligent machine will be built and that it will be
the last invention that man need make.
Good has captured the essence of the runaway, but does not pursue
its most disturbing consequences. Any intelligent machine of the sort
he describes would not be humankind's "tool" -- any more than humans
are the tools of rabbits or robins or chimpanzees.
Don't think that's at all what the article in question is discussing, but this was interesting.
The assumption I need to challenge is that "an ultraintelligent machine could design even better machines".
That assumes that better machines are even possible, that there's no point of bottleneck that cannot be broken, or that they are so much more intelligent than us that they too won't hit a limit pretty quickly to what they can figure out.
It also says it surpasses any man however clever, but doesn't say it can surpass the totality of man interacting and building on each other's. If you take the most clever of all man, they are not so clever as to surpass humanity's total ingenuity and invention. And it's not a given that this super intelligent machine, more clever than any one man, is able to replicate itself and work in tandem with copies of itself in a way that is as effective as how humans somehow form a dynamic system between themselves.
Lastly, and I know the premise assumes it's better at everything, including at building intelligent machines, but I think an ultra-intelligent machine could exist that struggles at building better machines. Depending on your faith, we might be the construction of lesser intelligent organisms eventually discovering and evolving into us, and yet we do not yet understand how to make a more intelligent human.
> The assumption I need to challenge is that "an ultraintelligent machine could design even better machines".
Would you bet your life, and everyone else's, to test that assumption?
Not to mention: you can scale up performance in simple ways: more data, more compute, and more weights. Those are the obvious ones, ignoring architectural changes.
reply