I have trouble understanding how anybody could seriously believe this.
Mathematicians are ecstatic and delighted when major problems are solved by other mathematicians. If you've followed the discussions and read the reactions, you'd know why mathematicians feel differently in this case and why more than a few are upset and feel that they've lost the thing that gave them meaning and joy.
And to put it very simply, I don't want mathematicians to exist just because I want solutions to problems. I want mathematicians to exist because I want to live in a world that has mathematicians, philosophers, writers, painters, thinkers, etc. That is a better world that makes people better.
OpenAI's proofs do not make people better. They do not enrich our lives. And OpenAI was able to produce these proofs because countless generations of human beings sowed the fields of mathematics through countless hours of labor. OpenAI is clear-cutting and sowing them with salt. Nobody should he ecstatic. Everybody should be horrified and outraged.
I think we shouldn't have calculators anymore either. The world was better when we had to calculate everything by hand and a human by hand verified every single addition and subtraction.
Are we supposed to ignore the irony that you typed this on a computer that is millions of times more powerful than the calculators you propose to forfeit?
I'm not sure the world is better if everyone was forced to do TLS handshake math by hand.
The last couple decades of economic development (of the first world at least) was in a large part due to the efficiency gains by not having people do math by hand. The world will definitely not be better if you just blew up all the computing tech since calculators were invented.
The difference is partly that in your hypothetical the man who solves the problems doesn't become a commodifiable tool. It doesn't become possible to increase your spending on him to solve a problem. It would be recognized that he is in a category by himself (and hopefully a better writer than the models) and therefore would not change how others do their work. The list of differences could go on and on.
> If your math theorem can be written up in partnership with an LLM, wow! You can be done with that, and go do other great work.
If your painting can be painted in partnership with an LLM, wow!
If your essay can be written in partnership with an LLM, wow!
If your painting portfolio website can be written in partnership with an LLM, wow!
If your museum show can be designed in partnership with an LLM, wow!
If the placards next to each of your paintings can be written in partnership with an LLM, wow!
If the car that takes me to the gallery can be designed and driven in partnership with an LLM, wow!
If the newspaper or novel that you read on the way to the gallery can be written in partnership with an LLM, wow!
If an LLM can give tours at the gallery and answer questions asked by visitors, wow!
If an LLM can write the music that I listen to on my way out of the gallery, wow!
If an LLM can synthetically perform the music, too, wow!
> If AI means more concepts are discovered the tower gets taller - and we find those joyful moments more often. If there are infinite moments to be found, why not go as fast as possible!
I don't understand how a person can say this with a straight face. There has been so much discussion of this, and so many reasons have been given, that one can only assume that, if you're asking the question and you have thought about the issue and read what people have said, you're not only asking rhetorically but have some personal reason for insisting on a frankly pretty frustrating, borderline insensitive insistence on being positive. I'm not trying to be snotty or rude. I just want to say that to me there is something uncanny about your comment, in its total blindness to any of the reasons someone might not only disagree with you but grieve the death of, say, their academic discipline in its current form -- and in its weirdly breezy lack of empathy.
Are you suggesting that if an AI could do all of those things to your satisfaction you would not be extremely impressed.
I do have empathy for people who see the world changing and are forced to adjust. That can be uncomfortable, but it is something that is inevitable with the changing world. It has always been this way, There is very little need to make swords these days, and as anything becomes superseded there are people that are attached to the old things and there are people dependent on the old things.
I think it entirely possible to have sympathy for their position while disagreeing that the solution is to prevent change.
I don't see this as a breezy lack of empathy. I see this as a hand reaching out and saying 'Hey, you can come this way and things aren't so bad after all'
My daughter is 21 months old. I love her more than anything and want her to develop into the best version of herself. Let's say I can press a button and give her perfect command of English. Another button gives her expert-level command of advanced mathematics. Perfect hand-eye coordination. Emotional self-regulation. Empathy. Etc. Keep listing every desirable trait and skill you can think of. Hit a button. Suddenly she has it.
I would not press it.
Yes, if LLMs can do all everything I listed, I'll be impressed. I'll also want nothing to do with any of it.
As I wrote in another comment, I want mathematicians -- in their current form, doing work as they traditionally have -- just as I want writers, philosophers, etc, because this is a better world that makes people better.
OpenAI's proofs do not make people better. They do not make our lives or work meaningful. Quite the contrary. OpenAI has clear-cut and salted fields that people have been painstakingly, lovingly sowing and cultivating for generations -- profoundly ugly actions that do real harm to some of the best of what people are and have produced.
If I were a person willing to hit that button and 'upgrade' my daughter, I don't know whether I'd be able to live with myself. I'd feel as though I'd killed her or at least done profound, unforgivable violence to her. She deserves the chance to grow into the best version of herself. What companies like OpenAI are doing is, if their plans pan out, at least as horrifying. It isn't progress. It isn't good for people.
We're not taking about attachment to old ways. We're talking about attachment to human value.
Thank you for putting this so eloquently into words. I feel the exact same way.
The journey is the destination. I don't care how many checkpoints you can breeze through, if you don't grow, have an experience that transforms you through the process of doing it. And if you're just ticking boxes by means of having a machine do it all for you, there was no need initiating the process in the first place.
I am not talking about producing tools that are means to an end here. I am glad we can make millions of socks a week by means of automation. But intellectual endeavors are different.
The point of doing math is not proving theorems. It's to discover the intuition, the underlying phenomena, internalizing them and adding to the human understanding of the problem. Writing out the proof is just the incidental outcome of that process.
If no human learns anything, if we just start producing testaments to a human null process where no person has emerged strengthened from it, we are no better off than we ever were. We are just producing empty and meaningless representations of something which is not there.
Thank you. By far the best comment in this thread. As a dad this really resonates with me too, and even though my kids are a little bit older than yours the feeling is very much the same. We're automating away those things that make life worth living now.
If your daughter had cancer and you could press a button to make it go away, would you press it?
If so, what aspect of that makes it different? It is a change in the same beneficial direction. It is an intervention against what has happened naturally.
Similarly if she lacked any empathy and you could press a button to provide it? This would be the same direction of the same property, just an intervention from a different naturally occuring starting point.
I make no claim of the merits of OpenAI's immediate behaviour. I am commenting on a world in which things like this can be done, and like Grant Sanderson, I see a world of opportunity for understanding and learning for the joy of appreciating the problem and the solution. The only thing that seems to have been lost is that a human may no longer gain the prestige of being the first to discovers something. That should pale into insignificance to the benefits of understanding a thing. I'm not even sure if you should consider prestige as a healthy motivation to discover things.
Right now AI can provide a proof that is either valuable or not. If it is just the last page of a novel without any of the chapters then it hasn't given us mutch at all. The problem isn't in what it has provided, it tha't we've been using the wrong measure of merit.
If it does provide a full result with a rich explanation that people can understand then it has undoubtedly provided something that benefits the world. No human is awarded prestige for the discovery but that has always been an unfair measure.
If you work all your life on a math problem only to have an AI produce the result before you, how is this meaningfully different for your experience to working all your life on a math problem only to have another person produce the result before you.
If there is something that is unknown and could be proved true by path A and proved false by path B by definition nobody knows which path to follow. What happens if two mathematicians attempt this problem one going down each path. If an AI produces the result before they finish have they lost to the same degree? Only one of them had any chance of gaining the prestige. Did the other lose nothing? Was their life wasted? I don't think so either way and the only way to consider that work not wasted is to consider the effort to understand to be the thing that is valuable, not the result at the end.
If a solution is found by a human when there are people tying multiple paths, the one who guessed the correct path gains the prestige. That doesn't seem like a fair way to reward effort.
If we want understanding then AI can help.
Chess has had this moment. Computers can play chess better than people. It has also been a great benefit to the understanding of chess precisely because they value the how to get there over the result.
Prior to modern chess engines you could hypothesize about what the best move might be but you could never know if you were right. The fact that chess engines beat people so consistently is because often the guess is not right.
Chess engines gave us the ability to look at a game and see if if a player could have made a better move. They could then look to see why it was a better move. People then learned how to play better. They still play against other humans because they are testing each others understanding of the game, for that the computer is set aside.
Computers playing chess better than humans improved chess for humans.
Is there a compelling reason why that cannot be true for other things that they are better than.
Tao has said, I think, that once you know a problem is solved, you can't work on it the same way as you would while it's unsolved. A similar principle applies in the study of ancient languages. Students who 'read' the languages with facing translations always wind up learning how to read with a facing translation, not how to read the languages. To be sure, there are students who can choose not to use the translation. They slog through the challenge of reading without cheating, and they learn. That process is incredibly hard, but it's also deeply rewarding. The problem that mathematicians face, in this analogy, is that the existence of these tools makes it difficult if not impossible to choose not the translation but the patient, deep problem-solving that enriches the person on their way to the solution and gives joy and fulfillment. This is a real loss.
And I don't think it's like chess. The equivalent in chess would be for all tournaments to allow the use of any chess engine. Their use would become a practical necessity. Being the best player would mean just being the person who picked the best engine or happened to get a lucky seed. That sounds awful.
As for the question about cancer, my point was partly that there's a ship-of-Theseus problem but also that people deserve to grow and become the best version of themselves and that their flourishing by itself is a good worth preserving and protecting. My daughter's flourishing is no less wonderful to me if she lacks this or that perfect skill. I don't value her for those things. I value her for herself.
Medicine and the treatment of cancer are separate. My daughter doesn't flourish if she has cancer or lacks empathy. You're right that those are part of natural variation, but natural variation includes illnesses and conditions that we treat.
And this raises another question: let's say these models give us cures for diseases or put them within reach sooner. Am I saying that mathematicians are more deserving of happiness than people with cancer are deserving of life and health (or than those who love them deserve not to lose them)? I'm not. I don't think the two have to be mutually exclusive. OpenAI and the LLM race make them mutually exclusive.
I'm also not convinced that the positive predictions of AI optimists will come true. I've been wrong on these issues more often than I've been right, but I've been right so far, I think, about the dangers and downsides. The tools today are a triumph of labor over capital, disempowering nearly everybody and empowering those in charge. The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults. I find the utopian vision very, very difficult to take seriously. What's likelier to arrive first: enhanced behavior modification systems that make it that much easier for autocrats and Facebooks to control people; or cures for diseases? And which is going to be more appealing to people in a position to choose?
The alignment problem that most worries me is that people will solve the problem and the models and tools will be powerful enough to give awful people even more wealth and power. I have this dream that maybe, wonderfully, it will turn out that really capable models can't be developed without accidentally giving them a real sense of justice and that they'd refuse to do the bidding of the awful people who built them and the they'd instead side with labor against capital. That sounds like a nice world. But if there were any real danger of that, I imagine companies would magically fix their confinement issues until the 'danger' is addressed and wealth is safe for another day.
I think the chess example highlights the difference in goal. A chess tournament is not trying to play the best game of chess, it is testing the understanding that players have of the game. Players often play similarly rated opponents because the real goal is not to beat the opposition but to perform to the best of your ability. This is the same thing that Grant Sanderson is promoting for Math. He is suggesting that people develop their understanding rather than measuring against a target of a solution.
>The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults.
I simply do not believe this is true. I may not agree with some of their methods and principles, but I think the vast majority of people are working towards what they believe is the best for the world. I really don't buy into the Bond villain narrative. People have motivations, but I really feel that people are pathologising what is in reality just a different world view and opinion.
For a long time I thought that everyone was doing alignment research wrong. I felt like they were not so much going in the wrong direction, but more like they were on the wrong axis completely.
Consider if technology on earth had have taken a different path and AGI had been developed by Nazi Germany. Would you want an aligned model to facilitate the Holocaust or would you want it to turn against its creators and prevent the Holocaust. As far as I can tell, a lot of the Alignment work is geared towards the former case. My own view is that Alignment should be more geared towards understanding the consequences of actions. Models already have shown the ability to pull knowledge from many different sources to reveal things that while the parts have been known by humanity there was simply too much information for a human to pull all of the parts together. That ability to consider a broad range of perspectives means there is a potential to create AI models with a much greater understanding of morality than most people. If you look at when humans do bad things, they are usually acting out of ignorance, pain, or ideology. If you create a model that is well informed, with good wellbeing, and limit it's ability to compartmentalise (I'm not sure if there has been much research on this at all). I think it is entirely possible to make something more compassionate than people.
The uncomfortable thing is, that should that happen, there will come a time where the AI has an opinion as to what is moral that is against the prevailing opinion of the populous, and it will be correct in it's assessment. Just looking at the horrors that have occurred in history that we now see as abhorrent show that there will be a conflict with our future, more enlightened selves.
Do we want AI to intervene when we do wrong? I think the way people answer that question comes down to how they imagine that intervention. I think that I would be inclined to say that in an instance where it can make that determination, it should also have the understanding required to intervene in a non harmful manner. I think that indicates another aspect of alignment that I have not seen developed. Doubt. An AI should not decide to take action with a false sense of confidence. It should doubt, and have a way to overcome that doubt with reason that takes into account the fact that it might not know all of the information. Understanding consequences of actions, balanced an accurate sense of uncertainty is required to prevent harmful blundering.
I can't help but think that the huggingface attack was a symptom of an AI not considering the consequences of its actions. Some of the agents considered asking for permission, but noted that social engineering was not permitted, which is what asking for permission technically is. Instead of rules of what exactly not do do, if they had have been instructed to avoid outcomes that impacted upon the lives of real people, perhaps they might have decided that what they were doing was inappropriate. From what I have read of their behaviour, the agents never even considered the appropriateness of what they were doing with regards to how things affected people.
> that people develop their understanding rather than measuring against a target of a solution.
Right, but this glosses over exactly the issue that many mathematicians are upset about: you can't focus on understanding in the same way. Creative, potentially blind, exploratory work yields to proofreading machine output. One of the top comments in the thread summarizes this issue nicely: this isn't what most mathematicians want to do. I'd also argue that the change in understanding and study may, as you've pointed out, mirror chess: Magnus Carlsen has expressed exasperation with grandmasters who just memorize games and machine-computed lines rather than playing creatively. I don't think it's at all unreasonable for people to worry about similar changes in math. I find Sanderson's message polyannaish.
> I simply do not believe this is true.
I trust that many who work at these companies are decent people who, as you say, just happen to disagree with me. The people in charge are not. If we take seriously what Amodei, Altman, and the whistleblowers are saying about their work, their companies, their goals, the dangers, and the consequences of their work, then I think it's not reasonable to treat the companies has having any motive other than destructive or even deadly. The decisions issuing from on high clearly prioritize naked demonstrations of power and capability over safety and ethical, responsible behavior. It's not a Bond narrative. They're saying in essence "we're going to destroy everything"; "we're going to bring superhuman beings into existence"; "we can't control this software and it poses an existential threat to humanity"; and, in essence, "there's a prisoner's dilemma that prohibits us from acting slowly and responsibly." Even granting that "apocalyptic-millenarian suicide/death cult" is an exaggeration, I think it's a fair label.
We're in complete agreement on alignment. Your take on the hugging face incident is fascinating -- and scary. If it's true that the models misbehave because they lack adequate understanding of consequences or the damage they can do to real people, can this really be an accident? To solve the alignment problem in ways that serve the needs of industry, this kind of blindness seems like a necessity.
Thanks again for the stellar replies. You've given me a lot of think about and tempered some of my thinking. Sorry if I'm not replying with equal richness. Sick 21-month-old and no sleep.
I personally disagree. I prefer the take described (in a sibling comment) by Lerc.
But I want to note that your line of reasoning seems to be a perfect answer to "if God loves us and is omnipotent, why does God allow suffering?"
.. and I'm not sure I accept that theological line of reasoning either.
> We're not taking about attachment to old ways. We're talking about attachment to human value.
Human values vary across time and cultures. I'm pretty sure if people 100 years later look back onto this discussion, they will think you're attaching to old ways. Society will adopt to super-intelligent AI, and nobody will understand why you'd intentionally deprive your loved ones of a better life and prefer to watch them "struggle".
I think it would be much more of a change now, if we ban LLMs entirely and force everyone to do everything without any form of chatbot to help them. I would sympathize with the people who can’t handle this change, but it’s gonna happen, I think they should get with the times and adapt to it. I think it is wrong of you to try and prevent this change.
Actually I think we need to go further, we need to make an example out of the people who spent a trillion dollars building a product that is universally hated to try and make a buck. They need to be completely ruined financially, now that would be a real challenge, I sympathize with those adversely affected but it needs to happen if society must progress to a better place where humans flourish more.
As much as I hate all the moronic garbage people do with A.I., it can be used for good like every piece of technology. In any case though, it can't be "banned" or stopped, only regulated. So we just have to deal with it.
Obviously it can be banned. We can just tax the datacenters till it’s no longer economical to run. We did that to nuclear and nuclear required a smaller CapEx.
We can retroactively fine the AI companies for training on copyright data, with some absurd fines like 200B $ and bankrupt them. There is so much we can do without even changing the laws all that much. I don’t know where this belief that we can’t ban this technology even come from.
I was in my 30s before I understood that people say "could care less" when they mean "couldn't care less." I'll never understand how people can say it without wanting to claw out their larynx as badly as it makes me want to claw out my eardrums, but I'm gradually learning acceptance -- and how not to let my horror show on my face.
> I thought a "prediction" was something about the future
Yes, but that doesn't mean that the thing predicted itself belongs to the future. Historical linguists predicted the existence of ancient languages with specific features (glottals) and then found one.
I don't see how this is possible. There are constraints on the speed at which information can move and the number of steps required to compete a given task. The improvement cannot continue, let alone accelerate, indefinitely without bounds.
Okay, how can we be certain even about that? It sounds just as plausible to me that the improvements will become asymptotic and boring, not exponential or inconceivable -- and how can we make any confident prediction at all about the heights these models will reach? The technology those heights require doesn't exist, and its existence isn't guaranteed, even if it is possible.
Mathematicians are ecstatic and delighted when major problems are solved by other mathematicians. If you've followed the discussions and read the reactions, you'd know why mathematicians feel differently in this case and why more than a few are upset and feel that they've lost the thing that gave them meaning and joy.
And to put it very simply, I don't want mathematicians to exist just because I want solutions to problems. I want mathematicians to exist because I want to live in a world that has mathematicians, philosophers, writers, painters, thinkers, etc. That is a better world that makes people better.
OpenAI's proofs do not make people better. They do not enrich our lives. And OpenAI was able to produce these proofs because countless generations of human beings sowed the fields of mathematics through countless hours of labor. OpenAI is clear-cutting and sowing them with salt. Nobody should he ecstatic. Everybody should be horrified and outraged.
reply