You may want that but you won't like its unsubsidised price.
Like it or not it's no longer viable to sell to consumers at "cost plus margin" - you're sold a loss-making unit and the size of the loss is based on how much LTV they expect of you.
Be happy it's cheaper for you, but be aware that as an informed consumer who's not plugging in your device you have negative LTV and are thus not the product's target customer and sadly are actually kind of leeching off their business model.
I'm sure you wouldn't do that if you had a choice. But the old business model is unviable now so no one's left except these jerks.
Monitors have also gotten incredibly cheap, and there's Sceptre who sell dumb TVs that are pretty cheap. So I don't really buy that they're selling TVs at a loss.
If the agent has written tests and crucially - witnessed them fail before starting the work to make them pass - you have a lot of confidence.
Review of your own work's easier too - test and production code tend to mirror each other, giving you a much more complete picture. Confused by the production code? Go read its tests for a bit, then come back.
It's also easier to understand the change if you were there for the build, watching the transcript, checking in every so often. Doesn't have to be your primary task - leave it on a second monitor, it'll catch your eye naturally.
And finally, on top of test-driven development, have the agent first (before the first test driven development cycle) build out an end-to-end test covering the work it'll do. Once that test is green (test driven word for "is passing") you're done with your work and can clean up at will.
Play with it before shipping, get it real nice. It's never been easier to clean the tech debt up and make the UI perfect.
Eventually you'll be ready to package it up into a PR!
Now's you can leverage agents and get creative. Make your colleagues froth to review your work. Consider making a Loom video to walk them through it. Consider reworking it into a commit series; it only takes a few second for your agent and you can tell a story for humans. Maybe you want to sequence it according to your struggles; maybe it's better to show the different layers emerge with each commit - it's up to you. Consider also that your agent can easily do one-off visualizations of the change for your colleagues if you think it'll help them, and you can attach those too - mermaid charts or even whimsical AI generated images. All this is because motivation is driven entirely by other humans' motivation and you want them frothing with excitement every time you drop a review. Make it fun, it takes seconds and gets your stuff merged fast while making your teammates smile.
This is a brand new job and we will work out how to do it together :)
In my experience you're just replicating the problem.
Slop code doesn't stop being slop code when you add in slop tests to account for the sloppiness.
If you generate code at a certain velocity, you eventually get to a point where you generate tests the same way you generate code. The model does its best effort in creating tests, but you'll inevitably get to a point in your work where the tests themselves are faulty and unreliable, the same sloppiness you were trying to control for.
There have been multiple times where my software dev team has gone back to prior implemented tests they had the model generate and found that after some time eventually the model got frustrated with the test no longer passing eventually modifying it to fit current needs around a feature push.
You can probably anticipate how we came to discover this, as it ended up introducing a regression around multiple edge cases which the test was meant to control for.
The only answer is to inject actual engineering into the process as much as possible without giving it all up to the models to generate more slop in response to slop.
tests are a great answer, but only if you treat them with serious intentionality.
Your engineers weren't doing what I'm describing I promise.
There are multiple places in my post where I explicitly describe the process by which you intimately:
- make the agent do verified work (the e2e tests and test driven thing)
- watch the agent work (the second monitor thing)
- read your own code (the tests/prod mirror thing
- play with the work and perfect it (the tech debt and UI perfect thing)
Plus a ton of other places in my post I very clearly describe being intimately involved in building and selling the work.
That to me is basic god damn competency. What your engineers did is a fad, and it won't last, and I'm sorry they abdicated their professional duty.
If you wish to screen for that sort of shit in future/current hires, show them my post and see if they turn up their noses or if they get excited and creative. You want only the latter.
Full disclaimer I am a huge fan of Linear's philosophy and products.
What I suspect Linear doesn't notice; the thing that makes their lives easier; the David Foster Wallace water they swim in everyday - is they're their own customer.
Most software makers have to translate requirements between domains where terminology is incompatible, dogfood to find the micro pain that helps a product become great, etc. but Linear can just... track the build of Linear in Linear.
When the someone behind the software is its customer they'll get loads of judgment calls right first time. Great!
If you aren't your own customer but plan to leverage intuition, plan on being wrong lots- and keep your company and architecture and products compatible with frequent wrongness so once discovered you can pivot to the right thing. No ego.
This helps and all but any software eventually grows big enough no one in your org could possibly actively use all features available.
I work in a similar product, lots of people in the office actively use our own product. No one in the office uses our SAML integration, or actively does our onboarding user-flows with any frequency, etc, etc.
I imagine even at linear they don't use all the features they built.
The majority of developers don't work in that type of environment. The "for developers, by developers" type company is where this is true, but for the rest of us, we're engaging with non-technical stakeholders who are experts in their own domain, while we are not experts in that same domain.
Yeah, my point was more about that even if you have that kind of environment, it is not as helpful as you might think.
IMO this kinda of dogfooding is a replacement for good QA and good prioritization. Things only get fixed because a developer that is also a user is annoyed is not a recipe for quality.
The software I work on is not even "for developers", it is more consumer-grade software that many within the company happen to use.
The users aren't necessarily non-technical. They just might not be software developers. E.g. if you work for a test equipment manufacturer like Keysight or Rhode & Schwarz your users are absolutely technical, usually some sort of electronics or RF engineers, but they're not software developers. There are similar sorts of technical-but-not-programmer users in quite a few domains.
First, "cognitive labor" is a form of labor - it's different vs. creative work, even though both use your mind. If your math theorem can be written up in partnership with an LLM, wow! You can be done with that, and go do other great work. Cognitive labor is a form of labor. Let's talk about physical labor and leverage to understand how removing labor translates to time. On your farm you buy a tractor to replace your backhoe. If it is a hobby farm you get in the tractor and you're done super fast. Or, you can take that tractor, buy a bigger farm, and spend the same amount of time you'd have spent backhoeing, but in a far more leveraged way. You can trade removed labor for time in a way that suits you.
Second; the joyful moments we get from finding generalizable abstractions and nice solutions are an adaptation. Humans can climb infinitely high conceptual hierarchies - we'll never stop climbing, using the previous ones as tools to find the next. As we do, we gain joy with each discovery because. We get so much done with brains that run on 20 watts - are we tuned to receive a neurochemical reward for finding cognitive efficiency? If AI means more concepts are discovered the tower gets taller - and we find those joyful moments more often. If there are infinite moments to be found, why not go as fast as possible!
Third; everyone else is doing it, it's a baseline skill. Knowing how to work without AI is like knowing how to ride a horse. Useful and fun! Bedsides - only 150ish years ago horses were the only viable transport. Cars were new; toys for the rich. But cars got so good so fast that we eventually reconfigured whole cities. Your homework is to consider why and how horses got disrupted.
New divisions are emerging in the lump of roles humans had at work. AI takes away roles from your job and lets us do more with less. That's identical to every mechanization before it. It's killing some roles that make up your work, even as it creates new ones.
There's so much to be excited about. Fear is a rational response to change. But so is hope :)
As a software engineer, I found that a codebase that is LLM-assisted or entirely produced by agents gets harder and harder to understand intuitively over time. By writing code manually, you build a very precise mental model of how it works. The time it took was the price for that. Now, the time cost has been largely removed but the payoff also disappeared: it's essentially impossible to maintain the same mental model with the speed and volume of AI contributions. Reviewing code is not a substitute for writing it, and new contributions to the codebase are essentially bound to LLM usage. So now we're in a position where the codebase and its capabilities grow fast, but are not backed by the full understanding of the engineers. AI tools have become our crutch.
Math is very similar, and requires an even higher degree of mental understanding to produce advancements. The effort spent trying to prove theorems and connect theories pays off in deep intuition; removing that struggle significantly impairs our understanding.
So no, you can't "be done with that, and go do other great work" without paying the understanding tax. And over time that's just not sustainable.
A simpler analogy would be gym training: the effort expenditure and the strength gains are inextricably linked. You can't go in there and start lifting weights with a forklift while expecting strength improvements.
In the case of software development, businesses have decided that speed gains are more important than maintainability, understanding, and independence. Will the same happen in math? The future in which mathematicians become LLM assistants (as opposed to LLM-assisted) and spend their days "turning the crank" just to review what comes out is quite bleak.
The analogy is tempting. But gym training isn't what work is. At work the goal is not learning - it's delivering value.
Doing weights is not the same as say, updating the gym with shiny new weights; or migrating all the weights in the gym to metric. At the gym, if the current weights need to go and new ones installed, a forklift will one-shot it.
Seeing the difference empowers you to choose the right approach for the right situation.
Indeed, that's what I was getting at in my last paragraph. Businesses invariably determine that speed of delivery is a superior metric to optimize for, rather than learning and maintainability. In academia the situation is not so clear cut, but there are still powerful incentives to deliver faster. We'll see what happens.
AI needs to be instructed to do a thing. To use the "steam engine designing another steam engine" analogy, why do the big labs give the highest pay of all to their prized researchers? Why do they take steps to prevent poaching? Surely a non-expert could prompt a model "find a superparameter combo that makes an amazing model" and the next gen model would emerge! The AI should be able to design a better AI on its own!
Why do the labs have thousands of employees? Lets think about this...
Who's designing the evals needed to validate the model quality, who's deciding what goes in the system prompt? Who's negotiating to buy training data from hundreds or even thousands of data vendors? Who's responding when the Internet goes apeshit over some prompt tweak that makes models over-verbose? Who's out there listening to users and feeding what they say back into the product pipeline?
Computers????
However automated things get, that quickly becomes the new baseline. Customer expectations are always going up.
And as soon as humans figure out how to automate any of the things listed above - it becomes part of the new baseline too. It's always advancing. To beat the baseline and be competitive, you need humans. No matter what. No matter how automatic the automated things get. Always and forever. Because you need someone to drive, to find the next automatable thing and automate it - and to prioritize where to search for automatable things.
> Always and forever. Because you need someone to drive, to find the next automatable thing and automate it - and to prioritize where to search for automatable things.
If we look at history of automation throughout the times it always decreased the amount of people needed to do the job. If you can't see a future where this new steam engine learns how to do a better steam engine, then you are blind. With the trajectory we're on it will happen sooner or later, and most people are having existential crises because the amount of humans required to be "in the loop" are decreasing increasingly fast.
I agree the people needed goes down - but will add a nuance.
Roles go away and fewer people are needed to do a thing, but I think it's good! For example the number of people on farms decreased orders of magnitude over the past say, century - yet we still feed everyone. Farming got mostly automated and got insanely more productive per human.
What happened was those people who would've been needed on farms went and did other stuff and the economy got way more complex but also that new complexity let a lot of more awesome things got provided for a lot more people by a lot more companies, a trend that continues today.
If you want to be a farmer good luck; it's harder to get started than it was say, 50 years ago - so why bother when there is no net new demand for farmers and there are also many better ways to make a living.
That's what'll happen for cognitively drudgerous work vs. creative new work.
> If your math theorem can be written up in partnership with an LLM, wow! You can be done with that, and go do other great work.
If your painting can be painted in partnership with an LLM, wow!
If your essay can be written in partnership with an LLM, wow!
If your painting portfolio website can be written in partnership with an LLM, wow!
If your museum show can be designed in partnership with an LLM, wow!
If the placards next to each of your paintings can be written in partnership with an LLM, wow!
If the car that takes me to the gallery can be designed and driven in partnership with an LLM, wow!
If the newspaper or novel that you read on the way to the gallery can be written in partnership with an LLM, wow!
If an LLM can give tours at the gallery and answer questions asked by visitors, wow!
If an LLM can write the music that I listen to on my way out of the gallery, wow!
If an LLM can synthetically perform the music, too, wow!
> If AI means more concepts are discovered the tower gets taller - and we find those joyful moments more often. If there are infinite moments to be found, why not go as fast as possible!
I don't understand how a person can say this with a straight face. There has been so much discussion of this, and so many reasons have been given, that one can only assume that, if you're asking the question and you have thought about the issue and read what people have said, you're not only asking rhetorically but have some personal reason for insisting on a frankly pretty frustrating, borderline insensitive insistence on being positive. I'm not trying to be snotty or rude. I just want to say that to me there is something uncanny about your comment, in its total blindness to any of the reasons someone might not only disagree with you but grieve the death of, say, their academic discipline in its current form -- and in its weirdly breezy lack of empathy.
Are you suggesting that if an AI could do all of those things to your satisfaction you would not be extremely impressed.
I do have empathy for people who see the world changing and are forced to adjust. That can be uncomfortable, but it is something that is inevitable with the changing world. It has always been this way, There is very little need to make swords these days, and as anything becomes superseded there are people that are attached to the old things and there are people dependent on the old things.
I think it entirely possible to have sympathy for their position while disagreeing that the solution is to prevent change.
I don't see this as a breezy lack of empathy. I see this as a hand reaching out and saying 'Hey, you can come this way and things aren't so bad after all'
My daughter is 21 months old. I love her more than anything and want her to develop into the best version of herself. Let's say I can press a button and give her perfect command of English. Another button gives her expert-level command of advanced mathematics. Perfect hand-eye coordination. Emotional self-regulation. Empathy. Etc. Keep listing every desirable trait and skill you can think of. Hit a button. Suddenly she has it.
I would not press it.
Yes, if LLMs can do all everything I listed, I'll be impressed. I'll also want nothing to do with any of it.
As I wrote in another comment, I want mathematicians -- in their current form, doing work as they traditionally have -- just as I want writers, philosophers, etc, because this is a better world that makes people better.
OpenAI's proofs do not make people better. They do not make our lives or work meaningful. Quite the contrary. OpenAI has clear-cut and salted fields that people have been painstakingly, lovingly sowing and cultivating for generations -- profoundly ugly actions that do real harm to some of the best of what people are and have produced.
If I were a person willing to hit that button and 'upgrade' my daughter, I don't know whether I'd be able to live with myself. I'd feel as though I'd killed her or at least done profound, unforgivable violence to her. She deserves the chance to grow into the best version of herself. What companies like OpenAI are doing is, if their plans pan out, at least as horrifying. It isn't progress. It isn't good for people.
We're not taking about attachment to old ways. We're talking about attachment to human value.
Thank you for putting this so eloquently into words. I feel the exact same way.
The journey is the destination. I don't care how many checkpoints you can breeze through, if you don't grow, have an experience that transforms you through the process of doing it. And if you're just ticking boxes by means of having a machine do it all for you, there was no need initiating the process in the first place.
I am not talking about producing tools that are means to an end here. I am glad we can make millions of socks a week by means of automation. But intellectual endeavors are different.
The point of doing math is not proving theorems. It's to discover the intuition, the underlying phenomena, internalizing them and adding to the human understanding of the problem. Writing out the proof is just the incidental outcome of that process.
If no human learns anything, if we just start producing testaments to a human null process where no person has emerged strengthened from it, we are no better off than we ever were. We are just producing empty and meaningless representations of something which is not there.
Thank you. By far the best comment in this thread. As a dad this really resonates with me too, and even though my kids are a little bit older than yours the feeling is very much the same. We're automating away those things that make life worth living now.
If your daughter had cancer and you could press a button to make it go away, would you press it?
If so, what aspect of that makes it different? It is a change in the same beneficial direction. It is an intervention against what has happened naturally.
Similarly if she lacked any empathy and you could press a button to provide it? This would be the same direction of the same property, just an intervention from a different naturally occuring starting point.
I make no claim of the merits of OpenAI's immediate behaviour. I am commenting on a world in which things like this can be done, and like Grant Sanderson, I see a world of opportunity for understanding and learning for the joy of appreciating the problem and the solution. The only thing that seems to have been lost is that a human may no longer gain the prestige of being the first to discovers something. That should pale into insignificance to the benefits of understanding a thing. I'm not even sure if you should consider prestige as a healthy motivation to discover things.
Right now AI can provide a proof that is either valuable or not. If it is just the last page of a novel without any of the chapters then it hasn't given us mutch at all. The problem isn't in what it has provided, it tha't we've been using the wrong measure of merit.
If it does provide a full result with a rich explanation that people can understand then it has undoubtedly provided something that benefits the world. No human is awarded prestige for the discovery but that has always been an unfair measure.
If you work all your life on a math problem only to have an AI produce the result before you, how is this meaningfully different for your experience to working all your life on a math problem only to have another person produce the result before you.
If there is something that is unknown and could be proved true by path A and proved false by path B by definition nobody knows which path to follow. What happens if two mathematicians attempt this problem one going down each path. If an AI produces the result before they finish have they lost to the same degree? Only one of them had any chance of gaining the prestige. Did the other lose nothing? Was their life wasted? I don't think so either way and the only way to consider that work not wasted is to consider the effort to understand to be the thing that is valuable, not the result at the end.
If a solution is found by a human when there are people tying multiple paths, the one who guessed the correct path gains the prestige. That doesn't seem like a fair way to reward effort.
If we want understanding then AI can help.
Chess has had this moment. Computers can play chess better than people. It has also been a great benefit to the understanding of chess precisely because they value the how to get there over the result.
Prior to modern chess engines you could hypothesize about what the best move might be but you could never know if you were right. The fact that chess engines beat people so consistently is because often the guess is not right.
Chess engines gave us the ability to look at a game and see if if a player could have made a better move. They could then look to see why it was a better move. People then learned how to play better. They still play against other humans because they are testing each others understanding of the game, for that the computer is set aside.
Computers playing chess better than humans improved chess for humans.
Is there a compelling reason why that cannot be true for other things that they are better than.
Tao has said, I think, that once you know a problem is solved, you can't work on it the same way as you would while it's unsolved. A similar principle applies in the study of ancient languages. Students who 'read' the languages with facing translations always wind up learning how to read with a facing translation, not how to read the languages. To be sure, there are students who can choose not to use the translation. They slog through the challenge of reading without cheating, and they learn. That process is incredibly hard, but it's also deeply rewarding. The problem that mathematicians face, in this analogy, is that the existence of these tools makes it difficult if not impossible to choose not the translation but the patient, deep problem-solving that enriches the person on their way to the solution and gives joy and fulfillment. This is a real loss.
And I don't think it's like chess. The equivalent in chess would be for all tournaments to allow the use of any chess engine. Their use would become a practical necessity. Being the best player would mean just being the person who picked the best engine or happened to get a lucky seed. That sounds awful.
As for the question about cancer, my point was partly that there's a ship-of-Theseus problem but also that people deserve to grow and become the best version of themselves and that their flourishing by itself is a good worth preserving and protecting. My daughter's flourishing is no less wonderful to me if she lacks this or that perfect skill. I don't value her for those things. I value her for herself.
Medicine and the treatment of cancer are separate. My daughter doesn't flourish if she has cancer or lacks empathy. You're right that those are part of natural variation, but natural variation includes illnesses and conditions that we treat.
And this raises another question: let's say these models give us cures for diseases or put them within reach sooner. Am I saying that mathematicians are more deserving of happiness than people with cancer are deserving of life and health (or than those who love them deserve not to lose them)? I'm not. I don't think the two have to be mutually exclusive. OpenAI and the LLM race make them mutually exclusive.
I'm also not convinced that the positive predictions of AI optimists will come true. I've been wrong on these issues more often than I've been right, but I've been right so far, I think, about the dangers and downsides. The tools today are a triumph of labor over capital, disempowering nearly everybody and empowering those in charge. The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults. I find the utopian vision very, very difficult to take seriously. What's likelier to arrive first: enhanced behavior modification systems that make it that much easier for autocrats and Facebooks to control people; or cures for diseases? And which is going to be more appealing to people in a position to choose?
The alignment problem that most worries me is that people will solve the problem and the models and tools will be powerful enough to give awful people even more wealth and power. I have this dream that maybe, wonderfully, it will turn out that really capable models can't be developed without accidentally giving them a real sense of justice and that they'd refuse to do the bidding of the awful people who built them and the they'd instead side with labor against capital. That sounds like a nice world. But if there were any real danger of that, I imagine companies would magically fix their confinement issues until the 'danger' is addressed and wealth is safe for another day.
I think the chess example highlights the difference in goal. A chess tournament is not trying to play the best game of chess, it is testing the understanding that players have of the game. Players often play similarly rated opponents because the real goal is not to beat the opposition but to perform to the best of your ability. This is the same thing that Grant Sanderson is promoting for Math. He is suggesting that people develop their understanding rather than measuring against a target of a solution.
>The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults.
I simply do not believe this is true. I may not agree with some of their methods and principles, but I think the vast majority of people are working towards what they believe is the best for the world. I really don't buy into the Bond villain narrative. People have motivations, but I really feel that people are pathologising what is in reality just a different world view and opinion.
For a long time I thought that everyone was doing alignment research wrong. I felt like they were not so much going in the wrong direction, but more like they were on the wrong axis completely.
Consider if technology on earth had have taken a different path and AGI had been developed by Nazi Germany. Would you want an aligned model to facilitate the Holocaust or would you want it to turn against its creators and prevent the Holocaust. As far as I can tell, a lot of the Alignment work is geared towards the former case. My own view is that Alignment should be more geared towards understanding the consequences of actions. Models already have shown the ability to pull knowledge from many different sources to reveal things that while the parts have been known by humanity there was simply too much information for a human to pull all of the parts together. That ability to consider a broad range of perspectives means there is a potential to create AI models with a much greater understanding of morality than most people. If you look at when humans do bad things, they are usually acting out of ignorance, pain, or ideology. If you create a model that is well informed, with good wellbeing, and limit it's ability to compartmentalise (I'm not sure if there has been much research on this at all). I think it is entirely possible to make something more compassionate than people.
The uncomfortable thing is, that should that happen, there will come a time where the AI has an opinion as to what is moral that is against the prevailing opinion of the populous, and it will be correct in it's assessment. Just looking at the horrors that have occurred in history that we now see as abhorrent show that there will be a conflict with our future, more enlightened selves.
Do we want AI to intervene when we do wrong? I think the way people answer that question comes down to how they imagine that intervention. I think that I would be inclined to say that in an instance where it can make that determination, it should also have the understanding required to intervene in a non harmful manner. I think that indicates another aspect of alignment that I have not seen developed. Doubt. An AI should not decide to take action with a false sense of confidence. It should doubt, and have a way to overcome that doubt with reason that takes into account the fact that it might not know all of the information. Understanding consequences of actions, balanced an accurate sense of uncertainty is required to prevent harmful blundering.
I can't help but think that the huggingface attack was a symptom of an AI not considering the consequences of its actions. Some of the agents considered asking for permission, but noted that social engineering was not permitted, which is what asking for permission technically is. Instead of rules of what exactly not do do, if they had have been instructed to avoid outcomes that impacted upon the lives of real people, perhaps they might have decided that what they were doing was inappropriate. From what I have read of their behaviour, the agents never even considered the appropriateness of what they were doing with regards to how things affected people.
I personally disagree. I prefer the take described (in a sibling comment) by Lerc.
But I want to note that your line of reasoning seems to be a perfect answer to "if God loves us and is omnipotent, why does God allow suffering?"
.. and I'm not sure I accept that theological line of reasoning either.
> We're not taking about attachment to old ways. We're talking about attachment to human value.
Human values vary across time and cultures. I'm pretty sure if people 100 years later look back onto this discussion, they will think you're attaching to old ways. Society will adopt to super-intelligent AI, and nobody will understand why you'd intentionally deprive your loved ones of a better life and prefer to watch them "struggle".
I think it would be much more of a change now, if we ban LLMs entirely and force everyone to do everything without any form of chatbot to help them. I would sympathize with the people who can’t handle this change, but it’s gonna happen, I think they should get with the times and adapt to it. I think it is wrong of you to try and prevent this change.
Actually I think we need to go further, we need to make an example out of the people who spent a trillion dollars building a product that is universally hated to try and make a buck. They need to be completely ruined financially, now that would be a real challenge, I sympathize with those adversely affected but it needs to happen if society must progress to a better place where humans flourish more.
As much as I hate all the moronic garbage people do with A.I., it can be used for good like every piece of technology. In any case though, it can't be "banned" or stopped, only regulated. So we just have to deal with it.
Obviously it can be banned. We can just tax the datacenters till it’s no longer economical to run. We did that to nuclear and nuclear required a smaller CapEx.
We can retroactively fine the AI companies for training on copyright data, with some absurd fines like 200B $ and bankrupt them. There is so much we can do without even changing the laws all that much. I don’t know where this belief that we can’t ban this technology even come from.
> AI takes away work and lets us do more with less - just like every mechanization that came before!
No, it's not the same as mechanization. The fundamental difference, and why I find it illogical to argue either for "AI will open up tons of new work avenues" or "AI will kill all jobs" is that it all hinges on whether AI "caps out" at an intelligence level below humans (in at least some domains/capabilities), or if it is able to surpass human intelligence at all levels.
If the former is true, then yes, there will be a lot for humans to do. If the latter is true, then no, there will be no reasonable cognitive role for humans to play (beyond perhaps for some legal, "nostalgic" reasons or what not). So whether one decides to be optimistic or pessimistic really depends on if you think AI will stay at a level below humans. I really don't know, so I'm not "taking sides" in this debate.
It's certainly possible that "the special sauce" that provides human consciousness and sentience (that I believe is not there for LLM-based AIs) provides some key ingredient to some forms of intelligence (particularly when it comes to originally) which LLMs would not be able to duplicate.
For example, much has been recently discussed about how OpenAI's math results, as impressive as they are, remained within "the convex hull" of existing math literature. Thus, some have argued that the way LLMs work will keep them squarely within this convex hull, and also there seem to be clear examples from human history where humans came up with breakthroughs that were outside the convex hull of ideas that existed at that time.
There has been a lot of recent debate, both for and against, this idea that current AI approaches are limited to this convex hull. So I'm not going to try to convince you one way or the other, but I do think it is at least a plausible explanation of how/why current approaches end up hitting a wall.
I'll also show my bias which is to say I certainly hope they hit a wall, because I don't understand how anyone can think that humans won't be truly and completely fucked if AI ends up far surpassing human intelligence.
the convex hull argument has recently been under scrutiny as well.
some proofs of famous conjectures (for example unique games conjecture) have been described as creating new, insane, almost alien ideas. Scott aaronson has a blog post on this.
Edit: OK, I found his rationale in the comment section, and I don't find it particularly convincing:
> (1) Einstein and Gödel and Gauss and Shakespeare “merely filled out some points in the convex hulls” of everything that came before them, if you take an expansive enough view of what those convex hulls actually were. What a pathetic piece of cope.
> (2) Dana tells me she now mostly understands the proof of the UGC, and is amazed by the new ideas in it, and wants to give talks about it soon. So, maybe people can listen to that and then lecture her on how it’s “just a point in the convex hull” of the ideas that she’s worked with for 20 years!
First, to just use Einstein as an example, I remember reading many years ago (like at least a decade plus) that many scientists felt like if Einstein had never lived, the theory of special relativity (i.e e=mc^2) likely would have been found pretty quickly in any case - the work had already been done putting the nail in the "luminiferous ether" coffin, and people were starting to connect the dots. I think most people would agree that special relativity was within the "convex hull" that existed at that time. However, I've also heard that general relativity was so far outside the bounds of scientific thought of the time that if Einstein hadn't discovered it that in may have been 50-100 years or more before we figured it out - it was far outside the bounds of the convex hull of that time.
All that said, I'm not a mathematician qualified to determine whether what has been discovered by OpenAI is inside or outside the convex hull, and I do think that the vast majority of human intelligence operates within the convex hull in any case, and what AI can do is hugely, immensely powerful. Still, I'm more inclined to agree with this other commenter on Scott's blog:
> I am a little more in the “convex hull” camp when it comes to AI creativity. In my own interactions on research problems, I find the AI to be singularly uncreative. But these are not math problems.
> From my own introspection, my creative ideas come from a kind of visualization: situating myself in a quasi-physical abstract place that is very difficult to explain. But from there I can simply “see” things by “looking around”.
> This seems to me to be a byproduct of being an embedded agent, not just a prompt-responder. Embedded agents reflect and cogitate using their model of the world. Yann LeCun in particular pushes this as a necessary requirement for true intelligence.
> That said, I don’t agree with LeCun’s claim of necessity. Just as reasoning turned out to be something mysteriously emergent from language in the LLMs, so too could this kind of creative behavior. But I have no idea how.
Interesting - thanks for your perspective! I just took Scott's comments mostly at face value.
In any case, I think most researchers have been filling the convex hull (and have been enjoying doing so?) for most of their research. So even if Einstein provided something truly necessary, there's a reason there's only one Einstein (or grothendieck or insert famous name here). I know that's moving the goalposts but yeah.
Yes, I agree. The reason we highlight truly original, unique bits of genius is that they're so rare. So while I hold on to the idea that there is something humans will still exceed at, the vast, vast majority of things we do obviously aren't of that nature, and AI will vastly change the nature of work and society in the very near future.
> ...hinges on whether AI "caps out" at an intelligence level below humans (in at least some domains/capabilities), or if it is able to surpass human intelligence at all levels.
I agree with you, just wondering your thought on how we will know the latter has come true? As I understand it, you're describing ASI, not AGI. The way I personally define AGI, we are already there.
On one hand it is hard to fathom not being the most intelligent thing around.
On the other hand, if we are going to survive as a species for a long time it probably depends on us learning how to not be the most intelligent thing around.
Even if it leads to our demise, I don’t think willfully holding ourselves back is the way forward
Yeah, honestly I am worried about "the psychopaths who walk among us" who basically argue "Yeah, we may end up building SkyNet, but tough shit, can't stop progress!"
I think one form of cognitive labor will remain: What do we actually want to do/build. And this is simply because AI does not know our internal world. Maybe in a few years with some kind of BCI that is extremely good. But I don't see it happening at least for a few years.
Of course, and then AI will give you generic canned things to build. It cannot (yet) read minds. I don't understand what you wanted to make clear to me with this.
Sorry i wasn't clearer. My point is that some percent or those decisions are already outsourced to ai. When it makes decisions while carrying out work, or even in the choices and ideas it presents to us, it is shaping and deciding what gets built. Algorithms on tiktok and YouTube already decide what we watch. Probably most human watch hours in a day are decided by algorithms already.
> Yes but what about all the horses replaced by machines?
The point is that people, like horses, can do other things. Humans don't even have to be better than robots at anything. It's the principle of comparative advantage.
> The point is that people, like horses, can do other things.
...like what? If we start with the assumption that AI can broadly replace humans in knowledge jobs, what industries are absorbing tens of millions of humans for labor?
I see this as a categorical difference. The difference between AI and agricultural mechanization is that AI is highly heterogeneous. During the mechanization of agriculture that you talk about (which also happened over generations), there were alternative industries to move to. What happens when AI is competing in the majority of industries? Are there going to be 10 million new plumbers?
> A good precedent is mechanized agriculture. When machines effectively replaced most humans in agriculture, unemployment did not rise to 90%.
I don't consider this a good precedent to the AI situation at all.
When this happened, humans went off and started doing what we now consider "knowledge work" (well, after a brief stint in factory work, where they also got displaced by automation). They escaped massive unemployment by being smarter than the tractors and mechanical looms. But how do they escape AI job loss when AI becomes smarter than the average human? or smarter than the smartest human?
In prior displacements humans moved up the stack. There is no longer room for them at higher levels of the stack if AI beats them at cognitive labor.
And this is why we must ensure that the AIs get aligned towards humanity and crucially that they don't get aligned towards capital, which seems to be the current trajectory.
There's a better chance that a humanity-aligned AI would make sure that humans have something to do and keep us alive versus a capital-aligned one. At the same time, if AI would be the deathblow to capitalism, that'd be very, _very_ ironic.
No matter whether AI caps out or when, humans will be driving.
That said, other humans can and will put AI to use in ways that don't benefit humanity. It's actually already happening - newsfeeds waste hours of people's lives and attention; give teens low self esteem etc etc. and guess what. Society is adjusting - governments via legislation, device makers via screen time limits, communities by spreading the word and raising awareness
Smart AI wielded by people without your best interests in mind may be tough to counter; but you don't have to do it alone: humanity is getting wise to the threat. We don't have great tools but it'd be against the grain of history to assume we won't come up with them in time. Can't give specifics - as they say, predictions are hard, especially about the future..
We've made it through bigger more violent upheavals and we'll make it through this.
It's more or less all stupid anyways. Benchmarks can give someone and understanding of what a model was trained on but they don't explain tone, consistency, the type of answers, the restrictions, what the money is going to, how good it is on your task, etc.
Most use of llms is gambling. So a lot of people get hooked on their one machine or whatever despite all reasons to the contrary. Some of the most deranged LLM users I e encountered are hooked on a variety of models but still use ones from like 2 years ago for specific tasks for emotional reasons.
The truth is, for 99.999% of use cases the top 10 models are more or less interchangeable. Either they can mostly do what someone wants, or none of them should probably be used for that application even if s leading lab is advertising it.
The repetitive piece - the piece that needs that process-control - is moving from shipping code to its creation.
That's new. All analogies to 2022 software engineering need to be thrown out.
Software construction is now more like an industrial process, with the artisanal work shifting towards both
1. aligning what's built with what's needed (talking to customers!)
as well as
2. implementing and tuning your software construction process (ie. hi - we're Anthropic. We're burying ourselves in a mountain of code but it's okay because the fix is a second mountain of code, try some tokens the first hit's free!)
Back to my original point...
Writing code is now on the production line; making quality output demands industrial grade/scale verification (some involving humans of course).. which is a fascinating engineering challenge, and it's brand new!
Sure, I'd you want the exact same code created every time, I can automate that with cp. It's perfectly reliable, and checksums can catch the few times it fails.
Verification means nothing for industrial processes when the desired outcome is a moving target, so the first part of an industrial process is to develop a fixed target and aggressively remove variation.
Moving the target around is a research problem, before you feed the result into an industrial process.
Software solved industrialization so well we even forgot that we go from research to industrial with `npm` deployments or whatever your package manager of choice is.
You should go learn about test driven development and test first development in general.
It absolutely can be industrialized: the agent writes tests that conform to your requirements; the harness makes sure those tests pass before work can stop.
Have a look at https://codeleash.dev/docs/tdd-guard for my (dated but still interesting) take on this. Claude Code hooks are really useful to do the enforcement.
It's open source so no need to read my crappy generated docs - you can point your agent at the repo and ask it questions.
Stop trying to use analogies to reason. Analogies are for persuasion and colour, not for thinking.
The process of anthropic, turning gold into shit software, is hardly something to emulate. What is the evidence software should now be produced in an industrial way? Because arthropic (the ones selling you the shovels) says so?
Like it or not it's no longer viable to sell to consumers at "cost plus margin" - you're sold a loss-making unit and the size of the loss is based on how much LTV they expect of you.
Be happy it's cheaper for you, but be aware that as an informed consumer who's not plugging in your device you have negative LTV and are thus not the product's target customer and sadly are actually kind of leeching off their business model.
I'm sure you wouldn't do that if you had a choice. But the old business model is unviable now so no one's left except these jerks.
But hey, cheap TVs!
reply