its not just math .. its math, science, engineering, medicine, tech .. ie. everything.
We need to ensure we dont outsource thinking itself - yes its great to have a lot of our knowledge encoded in books, and in the weights of LLMs [easier to access ]
Do we even 'have' a technology if it doesn't reside actively in some human minds ?
Some of math is handed down to apprentice mages, and not encoded well in books [ perhaps books dont encode intuition well, Ive had many cases where someone has explained something not in the textbook, a trick they learned from someone else ]
Apprentice mages only become mages by grappling with hard problems, asking questions, going on long detours, discussing/arguing with master mages, trying lots of tough thought experiments - all valuable "training".
Training using the AI might lead to a golden age where more mages learn wider and deeper, and the subject flourishes and more humans understand the tech.
The danger is using the AI makes us lazy - we dont grapple with programming syntax, draw math diagrams slowly by hand, check edge cases, try to prove things ourselves the slow way and build intuition in the process.
Another issue is ownership and access. Already RAM GPUs and soon CPUs are harder to get hold of due to competition with the hyperscalar datacenter buildout. This is an example of economic inequality effectively stealing resources from the mainstream.
Will math remain open or will it be owned by the frontier models ? With Math and science journals Elselvier almost had a stranglehold, bullying universities to pay massive bulk subscriptions - fortunately Arxiv solved some of that. Can I even own a copy of a music track, tv show or movie these days, when there it is so much more profitable for companies to rent it out pay-per-view or monthly ? There is every financial incentive to slurp up the whole of Arxiv, github and every science textbook into a frontier model and rent out access via pay per token. I no longer need to go to stack overflow to look up a simple algo, google can give me a pithy summary, so its likely that the LLMs will steal traffic from github, or github could be drowned in the volume of vibe code. Is there any point buying a textbook or going to university when the AI covers the material ? If no universities then we dont have a physical space for science and tech to be discussed. That activity might go online .. or may disappear altogether - we already have less socializing in person due to the rise of social media.
We have several "alignment" problems : humans arent aligned with each other, frontier models are aligned with profit-making which may differ from the common good, the race is on and so frontier models are more aligned to short term shareholder / VC value than to longterm benefit to humanity as a whole. These mis-alignments may have always been there, but are multiplied by the power and reach of new AI technology.
It might be better to view the near-future AI as an emerging intelligent alien species.
Regardless of whether it is 'conscious' or not, Roman Yampolski makes the point that AI may or may not have human-kinds best interests at heart [ hell, even humans aren't aligned ]
Can we control or trust a counter-party that has intelligence at a level near or exceeding the smartest humans in all domains ?
I dont even think we can even have a safe sandbox in which to run an AI .. and frontier model training labs are certainly not air-gapped.
I think 95% of the population wont get a choice of how AI is framed - the big money pushes how it will be framed for them, and most people have no training in a hard discipline like math or science, as a vaccine against that marketing push.
t
People will be corralled into trusting everything an LLM says as gospel, will be required to use if for almost all jobs, will have no actual choice in the matter - everyone who suggests otherwise is a luddite. [ Terence Tao is being painted as a luddite currently, for raising the concern of whether human-math will even survive ]
95% sounds too aggressive (where did you get this figure from???). Even now there are too many talkingheads on X and everywhere and share whatever they think.
It's ultimately up to yourself which frame to adopt.
It's not black or white. You can adopt it situationally.
Lee Sedol left Go world after losing to AI. Terrence Tao could leave and look for something else following his suit.
Voicing his opinion works and he can attempt to cope with AI changes with his own way. (be it training/education, etc or do something completely different)
Its probable that AI will roll out much faster than humans can develop that skill of using it in a way that increases not decreases human knowledge over time.
The current slew of excellent LLMs will probably cause a decades long dip in human cognitive ability.
If social media on smartphones had an impact of ability of students to concentrate, LLM AI will likely have a much greater negative impact on education.
There are trillions of dollars of investment pushing for humans to trust whatever an LLM says, and very little money behind encouraging people to tackle hard problems themselves and train themselves to think.
I have to say honestly, that AI looks like an apocalypse for human thinking [ Im glad Tao is warning people, I just dont think many people will listen ]
I've learnt more since the advent of LLMs than I did in the 10 years prior.
And it's not just me; that is most people's experience with LLMs.
But most of those people would also concur with your apocalyptic thinking.
AI is the apocalypse for thinking, except for me, only I am smart enough and experienced enough to maintain my cognitive ability while using LLMs.
The problem is that if you take away the LLM, all of a sudden you can't do any of the things you "learned". It's confusing familiarity with understanding.
I did actually use an LLM to teach myself photoshop recently for a fantasy map building project. Mostly I'd just ask it things like "I have a mask with 50/50 gray and would like an elevation height map to be an embossed layer on it. Where do I find that in the menus?" those sorts of questions.
I finished the project out feeling pretty confident in my photoshop abilities. Then I did a similar project later and realized I had to open up the LLM again as there were still a ton of holes in my knowledge. After this second time though I'm feeling confident enough on my own that referring back to the LLM is increasingly less common, as it's just a waste of time if I already know what to do.
but this is not how people use - or providers sell - LLMs. They don't ask questions about how to do small steps, they just ask it to do the entire project, finish quickly and learn nothing. If the workflow was many, tight iterations it would be closer to a tool than an automation.
"some" do not a system make. "Some" people succeed in the current educational system, but others would argue people succeed "despite" the system, not "because" of it.
It seems we're about to get a massive stress test of such a theory.
Based on my experiences seeing what people actually do with AI tools: very. If this was a situation where I was seeing higher quality apps, PR's, and community building I would have few qualms about AI.
Instead it's created swamps in the art community, choas and bloat in the FOSS communities, a stronger sense of mistrust with traditional search, a deepening of the Dead Internet Theory, more broken tools to scrounge through... where is this supposed "benefit" I'm getting out of this?
Bear in mind it takes very few individuals to create a LOT of this sort of low-quality AI content. That makes it hard to judge how many are using Agentic methods vs "only Advice" vs "no AI."
LLM is an issue (to most of us) when it replaces exploration, thinking and effort to aquire skill. Whether it's vibe-coding an app, or thinking through a math problem. Jury is out this will atually dimish the next generation of thinkers. But seems like a valid concern to me.
2. This particular case of LLM usage (to teach us photoshop) is replacing traditional tutorials. It can help people learn things. This is also valid to me.
The existence of 2, doesn't eliminate 1. Existence of 1 doesn't eliminate 2. There's no strawmans here. I also think it's fair that the promise of LLMs (and AIs), isn't to replace tutorials, but to displace and automate much more.
Look at the marketing. "Complete projects", "Build apps". "Get advice". How often are we seeing "build a functioning workflow and validation pipeline and collaborate in incremental stages" or anything boring to that effect?
As someone who spent a lot of time supporting users and understanding their challenges with these tools, there's a lot of can you do X vs how can I do X, at least a 10x if not 100x difference.
IE: Model do this for me vs teach me.
Enough that I know people who have built businesses purely training people.
There was a brief moment where programs had decent built-in help you could search to easily find where stuff was in menus. Didn't need a billion parameters or a giant faraway data center...
Or, better, the global menu search. Intellij's shift+shift shortcut is great.
This knowledge has always been present in the product’s documentation. What you needed for learning is a chatbot, not general-purpose probabilistic automation which acts as a conversation simulator.
The learning is at that level of abstraction, so yes taking away the abstraction prevents will stymy you.
Just like if you pressed a button to wipe out our main language stack at our job, yeah none of us would know how to do anything with assembly or machine code.
> Just like if you pressed a button to wipe out our main language stack at our job, yeah none of us would know how to do anything with assembly or machine code.
All the devs would spend a couple weeks learning assembly and go on coding. And all the non-dev vibe coders would be unable to continue.
Yeah but then it goes the same way with LLMs? Just spend a few weeks delving the stack and then you go on coding. Learning a codebase of enterprise-scale compiler-generated assembly sounds like a much bigger nightmare than fixing up ai generated code.
*Many* devs would be completely incapable of learning assembly. Heck, many would be unable to learn C.
Likewise, *many* vibecoders would absolutely fail to adapt, and some would adapt, with this being their first experience. Just think of how many people started coding by using old gamemaker tools or scratch, or playing silly games. It is not the same as an LLM, but the idea that it is a gateway is definitely there. I wouldn't call LLMs "programming", because they're doing more: they're building products, which involves a lot more than programming -- hence why the skills to really thrive with LLMs are not just those of a good developer or engineer.
But weeks? I'd say less than 10% of the devs I've known my entire life would develop properly in assembly (and I'm being quite generous). Memory management, pointers, registers...just to name a few.
I think far too many people fail to realize that, with LLMs, the brain of a good problem solver, especially one with engineering background, gets much better results than someone with different characteristics. Similarly, the mind of a good product developer (and even someone with good UX inclination) does a much better job and building usable products.
This idea that LLMs are doing all of the work is frankly mind-boggling to me. It must stem from vastly different experiences of what LLMs do or whomever is using them around who says these claims. Obviously they do a lot of work, especially in the development prototyping, architecture and even ideation parts. But they only do (almost) "all of the work" in very simple or bad apps.
I've seen loads of terrible vibe-coding jobs: UX is terrible, bugs everywhere, clearly no consideration at any point for architecture (and thus the product is barely holding on). And I've seen the opposite: products that feel and act like premium stuff, because they were built by people with true love for what they do, sometimes even with a very good problem solving mind to add to that (leading to a not-so-terrible-albeit-still-barely-holding-on-but-thats-clearly-good-enough-for-the-product architecture). For every good app there are maybe 100 lovable apps that all look the same and break the second you do the wrong sequence of clicks, but the good ones do exist.
It's a tool that enables mediocrity to exist where it didn't previously, but it also empowers so many people in incredible ways. Still, you definitely need to be good at using the tool, and it definitely doesn't just happen overnight.
Isn't OS still required for a CS degree? Didn’t we all write our own kernel and virtual memory system in MIPs assembly code? We’ve been here before, I’m sure a lot of future AI coders will be forced to learn the abstraction stack like we did in a formal CS program.
As the field of "computing" develops more and more high level abstractions, that necessarily reduces the amount of time that can be spend on lower-level stuff. These days I think it's typical to have one or two courses that cover OS/embedded/assembly/compiler type of topics, but that's the bare minimum for awareness, not near enough for actual learning. On the spectrum from low-level to high-level, CS sits just below SE. Computer engineering is the field/degree that gives people actual skills in OS-level programming.
I've seen "software engineering" degrees where they teach some high level coding and how to manage tickets and this sort of stuff.
0 mathematics, 0 algorithms, 0 formal languages.
Universities are pumping out "professionals" like crazy, plus nowadays they are replacing courses with new courses about AI (which might be machine learning in the best case or just prompting in the worst case).
The result is that you need to go and ask what courses where in a degree, and even then it's not really a guarantee of anything (I've met master student with bachelor in computer science who were completely unable to find the smallest number in an array).
The courses might have assembly and stuff on paper (in many places not even that) but in most cases you write very simple toy programs, nothing like real-world stuff, reading compiler output or writing larger asm programs.
And yet: I have not written assembly since school except for one Advent of Code challenge, and I expect it would take me quite a while to pick up a productive level of facility with it.
I learned MIPS assembly and there were brief references to x86 assembly, and I know most of my friends would do fine with assembly, but, alas, most engineers I've known barely even know what memory management is, so being able to create proper abstractions with assembly would definitely be out of their league.
It's vaguely similar to how JS and Node enable so many people to flood the market without really knowing the underlying technology. Many of those would definitely not pick up assembly (which is the entire point I was trying to make)
Hell, I did a Physics degree, and I had to learn assembly (the NS 32k), which wasn't a huge issue alongside the 6502, Z80 and m68k that I already knew.
Now retired (come join the dark side, we have cookies) but still a few years away from what people would call "retirement age"
I need to retire so I can really tear into my massive project backlog. :) I'm really enjoying teaching, but it's unclear if this position will exist 3 years from now.
I think some still do. But writing a few lines of code in some computer architecture course isn't really enough. It's not just the language, it's the libraries and not understanding how things work under the hood.
I remember back in 2014 we had a class in assembly (we used x86 assembly running in DOSbox). One of the projects I most vividly remember was writing a simple paint program controlled using the arrow keys.
I imagine that was x86, which was relatively easy but also horrible. Very few people are writing assembly today because you'd need to be able to do x86-64 SIMD, ARM64, and maybe RISC-V, which are all quite a bit more complicated.
MIPS, because of the books provided for computer architecture (Hennessy and Paterson) were written by the MIPs designers. No one I knew of learned X-86, the Dragon compiler book however had all its examples in VAX assembly.
Everyone else is saying MIPS, but we learned Motorola 68k Assembly. Which I suspect was mainly because we had a bunch of dev boards for it. It was fun starting off by toggling the lights, using the switches to make a primitive binary calculator, before leveling up to making programs than communicated over the serial port.
We learnt MIPS assembly as part of the processor design class, where we built a DLX processor & emulated it running our code, and then we learnt x86 assembly as part of a class on debugging & reverse engineering (which is also where you'd usually have the good old bomblab assignment). That was in 2014.
To add to what you're saying, that's also ignoring that having the time or ability to build really complex things in assembly is harder, higher level languages making things easier also enables people to do more complex things
Assembly is simple. You can learn it quickly. It’s writing a large program in it that’s hard. How roller coaster tycoon was written in it boggles my mind.
The notion that the average developer would recover from the removal of modern high level language abstractions and frameworks, learn assembly, and go back to their regular coding velocity is patently absurd.
But that’s the whole problem. Almost no one has experience programming in the large with assembly. It’s far more difficult than building the simple toys built even in good university programs. It would be a total shit show and probably upwards of 80% would quit or crash out.
but even the smallest modern programs would be monsters in Assembly, so it's simplicity doesn't help much, and you're juggling a lot more than 3 bean bags.
Which one is more at risk of having the wipe-out and cost levers: your FOSS language/compiler or your paid third party LLM?
Sure, there are similarities, but while we're figuring out a FOSS solution that will run on your machine and we'll have access to more than the weights, the abstraction ladder isn't as equivalent as we think they are.
> Which one is more at risk of having the wipe-out and cost levers: your FOSS language/compiler or your paid third party LLM?
Neither is likely to go away. I do not expect it will ever be difficult to access an LLM in the forseeable future, barring massive legislation + enforcement to entirely ban them (I can run several capable coding models locally)
Nor is it likely that for developers at most companies will not have access to at least some LLM of comparably intelligence to those we have today (more advanced models may be restricted or too expensive) - right now large parts of the world economy are retooling to expand ai inference volume, not decrease it
This seems like "what if you don't have a calculator!" - and like with a calculator, there are benefits to learning things without the tool aid of course, n(including developing a sense for when things are wrong) but professionals are using the right tools for the job
>I do not expect it will ever be difficult to access an LLM in the forseeable future
If you scope LLM's to "frontier models", I do expect it to be an ever-growing moat. Maybe not if you work at BigCo who will pump out $X0 million/yr to gain access to it, but definitely for the singular use cases of people navigating to Claude.ai and asking a quick question. That's assuming the companies stay afloat long term to begin with.
If you open it up to open-weight models, then I question how many this will filter out. Will it be a tool valuable enough for people to overcome the large starting hump like Photoshop? Or will some future company come in and streamline the UI/UX to make them as intuitive as possible? Those are the truly interesting questions in my mind.
How feasible is it to have a local, FOSS, encyclopedic LLM for question answering and basic help with common technical tasks? I feel like that would be enough for me.
Nobody says it does. However, if I am solving big business problems and making enough to justify a private cook, why wouldn't I do that to optimize and focus on my area of expertise?
I don't want to be a code monkey. I want to talk to business people and come up with solutions. I don't care if LLM writes most of it or a junior dev...well I do LLM is better.
Ironic that the parent poster should use the phrase "code monkey" since most would likely agree being an "LLM prompt monkey" is an even lower rung on the ladder than that. The business people can surely find a LLM prompt monkey for much less salary than what a code monkey gets so I'm not sure anyone would want to embrace that future. In fact, if the AI gets good enough, the business people won't need the LLM prompters at all; they'll talk to the AI themselves.
Business people already talk to LLMs, and people who know how to code hate it.
I haven't written single line of code since Dec but I've been putting a lot of focus into systems design and making sure I understand how all the pieces of our system work together even if I don't know how the individual parts work.
The amount of times I hear my (fairly technical, mind you!) boss say "why can't we do X, Claude says it's simple" is pretty baffling, and my coworkers and him will have hours long debates about it.
Lol. I think a disconnect that engineers often have with product people is that product people want it to be done now (right or not) and engineers want it to be done right, which obviously takes more time. But I think a good engineer knows how to get it 80% of the way there quickly, and knows which 20% is ok to sacrifice or take on as tech debt.
Alas I'm not one of those engineers, but I'm fortunate enough to work with some.
There is a lot of that, but there is also the fact that you can get lulled into a false sense of "it is working" in general the motto is 90% of the work in 10% of the time and 90% of the time to get the 10% more done, and the devil is always in those 10% that often hinge on decisions made during the 90% phase.
It also depends on the requirements, and what the actual thing you want done. For software, sure you can probably patch it and there are some inconveniences you can live with depending on the clients requirements, for hardware... a little bit tougher... But it still depends on what is being done and what are the constraints, as long as people are reasonable, usually, it works out... Except that people rarely seem to be reasonable...
The issue is that the business problem you're solving is the problem of producing food. Why should you be kept around as the middleman, instead of hiring the chef directly?
No it hasn’t, middle managers are the ones telling everyone below them to use AI as much as possible, even if it makes no sense and screws things up. AI is a middle manager’s dream.
Yes, this is what I meant. There hasn't been much news on it for some reason, but in Microsoft at least they're getting pushed out of their role, either to some more direct role, or out.
I'm sorry to say but the business people don't want to talk to you, they want to replace you, that's what the LLM is for as far as they're concerned. Currently you're just filling the role of beta testing that replacement. The fact that you're so happy to throw away your fellow devs in the hopes of getting in bed with those people just makes this whole situation quite sad.
Maybe just get an MBA or something, I doubt its any safer from LLMs but thats clearly what you want to be doing
You can learn to run a restaurant without learning how to cook. That's part of the grief, fewer cooks and more restaurant managers. Some people like being cooks.
The ironic thing is the head chefs of restaurants do almost no cooking. They order around the sous chef, who orders around the kitchen staff, while being focused on the big picture.
In case you've never seen it before, you can watch how a burger's made (in case of LLMs anyway) and eventually replicate it. You can also inspect the recipe after the fact to learn.
But if I have to intervene at every fricking point along the way from frozen patty to cooked burger, if I have to tell the worker not to drop the patty on the floor and continue cooking it, which dials to turn on which machines, how long to leave the thing cooking at this stage, if I have to have the holistic knowledge (if not the details of the PID loop for the heating element), then it's a lot more than "calling for a burger".
"I" wrote a compiler with help from Claude - it's not the first compiler I've written, probably won't be the last, but it's a damn sight easier to tell the machine to implement LICM or to specify the requirements of the SSA invariants rather than actually do it myself.
Here’s the full and complete source of a GPU-powered Mandelbrot program:
#use "Stdio.xc"
#use "Par.xc"
#import "../src/include/bench_time.xc"
#define W 2048
#define SIZE (W * W)
i32 main(void)
{
// Eight runs: the first carries one-off costs (a GPU builds its kernel
// then), so the figure is the best run; the first is printed too. Auto
// decides after its fourth (two on each device), and uses the rest.
u32 total = 0;
i64 best = 0;
i64 first = 0;
for (u32 rep in 0..8)
{
total = 0;
i64 t0 = bench_now_us();
par mandel :reduce(+ total)
{
for (u32 i in 0..SIZE)
{
float cx = (float)(i % W) * (3.0f / 2048.0f) - 2.0f;
float cy = (float)(i / W) * (3.0f / 2048.0f) - 1.5f;
float x = 0.0f;
float y = 0.0f;
u32 k = 0;
while (k < 256 && x * x + y * y < 4.0f)
{
float xt = x * x - y * y + cx;
y = 2.0f * x * y + cy;
x = xt;
k = k + 1;
}
total = total + k;
}
}
i64 t1 = bench_now_us();
if (rep == 0)
first = t1 - t0;
if (rep == 0 || t1 - t0 < best)
best = t1 - t0;
}
printf("%u %lld %lld\n", (total + 500000) / 1000000, best, first);
return 0;
}
Just one line in that source code is what turns it from a bog-standard C-like program into a GPU kernel, automatically tracked, scheduled, and all data hazards transparently managed.
On my M4 max, that one line of code takes it from 481ms to 1.6ms. All ‘par’ does is say “this next block of code, call it “mandel”, is data-parallel, and for the ‘total’ variable all execution units should combine their output to that variable using +
No huge GPU boilerplate, no stream management for data transport from cpu to gpu and back, no memory barriers, no hassle. Just “hey, do this on the gpu”.
GPU blocks are bound at runtime using the best design for the hardware on the machine, obtained by measuring it (if it hasn’t been measured before). If it has, the cache is read. The user can always override, which is why ‘par’ blocks can be named.
One of the nice things about that is that if you find the bit you want to accelerate is too small to be useful, say the data-transport outweighs the computational speedup, (a) the runtime computer will figure that out and (b) you can hoist more code into the GPU if that's an option, to give it more work to do. The language is the same for both CPU and GPU so it's transparent...
The other thing is that the compiler will produce MSL, PTX for nvidia, SPIR-V and WGSL versions of the 'par' block, and since it's bound at runtime, you get support for pretty much everything.
What I'm looking at right now is multi-GPU systems. Consider a GPU to be a 'worker thread', and for each thread overlay the time spent uploading work units, the time spent computing, and the time spent downloading work results to the host. Now on Windows you only get 1 asynchronous copy engine for "consumer" GPUs like RTX {3|4|5}0x0 so there are limits, but MSL lets you have more I think and I haven't looked at AMD GPUs yet.
The thing that most people latch onto is the speed upgrade (419ms -> 1.6ms) but the real story here is the simplicity - using any dedicated language (CUDA, MSL, ...) you write a load of boilerplate and you have to worry about data hazards etc. I think of this as similar to when Apple introduced ARC (Automatic Reference Counting) - prior to that, engineers were constantly having to manage their memory manually, and there cough were a lot of errors. ARC changed the game, and I think xc is "ARC for GPUs".
I don't think this is true. The LLM provides confidence that it won't be a tremendous waste of time to try something you don't already know how to do. But eventually it gets tedious to consult the LLM all the time so you stop doing so. That initial "couldn't hurt to try" boost is worth a lot.
LLMs have also pushed me to learn a lot, but I don’t find the shape to be terribly different than before. I rarely remember the exact sequence of commands or flags or syntax. What my learning was useful for before and after LLMs is to 1) build an intuition of how something behaves and what it should do and 2) expose me to non-obvious possibilities that I wouldn’t have come to on my own. With those two things in place, the next execution without the LLM is a docs search, same as it always was (for me maybe some people are better at remembering this stuff).
What’s different after LLMs for me is that there are some things that just don’t care about and I’m fine if they are a black box that the agent handles for me. I didn’t have that luxury before.
No, the real test is if you can talk coherently about AI acquired knowledge to someone else. And there’s a lot that I’ve learned that I can now talk to other people about to coherently no agent involved.
I recently spent a day coding by hand. I was able to build a project, about 200-300 lines in python. Took me half day. But I could still do it. LLM would have done this in less than half an hour.
After this I realized why I personally choose LLM to write code: it is simply more convenient.
As a parallel, once you learn how to drive you stop walking, as for the primary mode of transportation.
Where do these weird car analogies come from lately... No, walking is still my primary way of getting around. That analogy isn’t as universal as you seem to think.
I'm walking as my primary mode of transport. Am abke to drive, live in city centre can reach my common destinations by foot. Public transportat for things bit further away. Car at low to mid distances is more hassle than it's worth due to parking.
Are you suddenly a history expert because you listened to every Hardcore History podcast? Better to read 1 book on a subject than listen to hours of podcasts. I gaurentee more is remembered from the book.
Just that there is still a big latency. Its smaller than finding the correct information on the web, but you can do so. Especially with MCP. The question is when this goes over into the physical world.
Watching Youtube videoas about anything that requires skill does the same.
Wouldn't you still be able to learn from the resources you created with LLMs? I've not found it that difficult to figure out how an LLM coded a program or website when requested, or to manually make changes based on its output.
If I use an LLM to create something in a programming language I'm not an expert in, I still have a reference point for later use. And that reference should drastically speed up the process of learning how to use said language myself later.
you can "learn", but it is the same and when you start to learn coding, or do homework, you read the solution and "aaah, I get it", but next time you still won't be able to do it unless you've done without reading the answer before.
reading code is not the same as writing code, the second one is much harder.
I have sat down and figured out how to program an Apple //e this year for fun. It is possible. It was quite instructive on how much has been added to the idea of what a computer is these days.
> if you take away the LLM, all of a sudden you can't do any of the things you "learned"
What? That is absolutely not true. I've been using LLMs for language learning, math/physics, clarify home improvement best practices that I now regularly think about, etc. and that knowledge is not LLM-bound at all. Instead of acquiring from a regular textbook, I acquired it from a model.
You're in a sarcasm thread. Distillation is nothing like stealing. Distillation is what's gonna save us from mistreatment by would-be monopolists who "stole" everything they "own" in the first place.
And it depends on the stock market continuing to go up relative to inflation (real inflation, not measured) and the labour market being strong enough to employ people. Which in turn relies on the USA's exorbitant privilege, which relies on all other western countries generally trusting it.
I think it's like driving without navigation systems. Yes, you have to make an effort to do it, but then it's fine. You don't have to use LLMs every minute of the (work)day.
This is why local LLMs is THE most interesting thing at this moment in time; the distribution of the tech is fundamental to it's long term viability. But now that this is happening in very solid fashion I would wager that LLMs are technically akin to most things a modern humans use during their daily life, utterly beyond their capability to understand, yet totally practically useful - this trend has been going on for hundreds of years at this point.
As an educator (of working professionals / continuing education), I see first-hand the range of AI use amongst the population (and even my population is not a representative sample, as it's people who are intrinsically geared towards being interested in building with AI), but your experience unfortunately probably reflects around ~30% of the population. The other third I'm slightly less hopeful about, but they might have a chance merely surviving, and the latter third I'd place in the 'full cognitive offloading / total abdication of all thinking and learning to machines' - for them, as the kids say: ngmi. Even within me as a single individual, I can sense the times AI is so tempting to offload cognition, and other times where it has been like a 100x speedup in my own innate skills and abilities. The future has come on quick, and we are barely getting to adjust healthily to it.
I was thinking the same thing. I see, and have seen, what the poster you responded to is saying. But, I think there likely is a bit of selection bias amongst folks who would be using an LLM to learn in the first place because folks who would use an LLM to learn will likely associate with others who would do the same, esp in a workplace. Ex: cars or computers. A small subset will tinker with them to learn how they work or improve them, but the vast majority just use them to enable other goals. But, it is still important that the small subset exists and is curious enough to learn how they work. And the folks who would tinker with a car likely associate with other folks who tinker with cars, and the same with computers...
Remove LLM and 80 pct of people will have to jump to start. LLM assistance is a huge dependence on something owned by few. And although I use it daily since Nov 22, I don’t see any big progress in productivity. Yes, I have done a lot of projects (that I can’t remember), I see a lot of sites made without human touch, I have heard fairy tales about llm’s breakthroughs to internet from a cage.
What I see that everybody is somehow a blacksmith now without knowing how to forge iron. And we call it progress.
I am optimistic, and I am not against LLM. I like a quote from OP’s article:
> One of my colleagues used to say, "AI is a mirror." I find that insightful: it reflects who you are, including the curiosity, judgment, and habits you bring to it. The questions you ask and the standards you apply to its answers matter.
My first project with it was in Jan 22, back when it was just sentence completion. Really hard to implement practically.
But in the past year the production boost is just insane. LLM assistance took away so many of the barriers that prevented me from executing on ideas.
I've also learned more than ever before, although it is admittedly getting harder to stay involved when the models are becoming more and more reliable.
What I have not figured out yet is why most developers around me do not share this effect. It's like it mostly works for a very specific subset of people, that do understand code, but it's not their main focus.
Just take a look at what is happening with the M-Wave FM1. Stating those devs would be lost without Ai won't stop the tide. It's not going to go away.
> Remove GPS and people will have to figure out how to read maps again.
It's not GPS, but turn-by-turn navigation. Remove that and people's spatial understanding of their environment improves significantly, and this improvement persists even long after the experiment, and they'll even be able to use turn-by-turn navigation better.
Which aligns with Anthropic's reports that people are able to use Claude better if they understand their own codebase better, and that increased Claude usage leads to reduced understanding of the codebase.
When society replaced walking as the primary mode of transportation with driving, people started going to the gym to keep their body in shape.
We should probably be training spatial perception (replaced by GPS), understanding of algorithms and data structures (replaced by coding agents), too.
Counterpoint, lots of people learn to navigate the areas where they spend their time by following GPS directions repeatedly, and can drive regular routes without them (though they're still useful for routing around unknown traffic issues).
I understand that it feels that way, but this has been widely studied and it's just not the same.
When you repeatedly follow directions, you'll learn how to take that path.
But when you repeatedly read a map, or walk an area on foot, you'll also learn the roads and areas you're not taking nearby, and learning the spatial relationships between these places.
A lot of learning happens through the mistakes we make, because they teach us a lot more about our environment. Whether that's a physical landscape or a codebase.
When you search for a bug, you'll look at maybe one or two files that are related to that bug, but also 10-20 files that aren't. You won't consciously remember the content of those 20 files, but your mind is building a map of their spatial and logical relations, which helps you the next time.
There is a giant difference in retention and learning between following even a printed turn by turn guide and using a modern gps.
This is anecdotally obviously as well as proven by studies. You don't learn the route or the neighborhood when you just "mindlessly" respond to "turn left" directions.
And much like society has rebuilt itself around, for example, the cellular phone and even the car, it’s rebuilding itself around LLMs. Yes, now you have to buy a thing you didn’t in the past. A bunch of capitalists have found a new way to build capital. I guess we get some value out of it too, but once it becomes built-in it can be hard to see it.
Fair point. How many people know OpenCode, Pi? How many pct of ppl code? How many pct of companies buy their own hardware with GPUs? It is similar situation e.g. as Salesforce vs Odoo. Imho 99 pct of companies choose SF, not Odoo.
People don't know about those because the options owned by few are cheap (subsidized by VC) and easily available.
My worry is that if we need to pivot to those open options, we'd be so far behind in infrastructure, and no one has the hardware to run the open models at their home/office.
And the overwhelming majority of LLM users will never use them or even understand what they are or how to set them up. Those people will remain dependent, and we shouldn’t ignore that.
>I've learnt more since the advent of LLMs than I did in the 10 years prior. And it's not just me; that is most people's experience with LLMs
My own usage too. Yet I believe that your environment is biased towards ambitious adults aligned with your values, who want to learn things. I don't think it's representative of world as a whole, and I'm even further worried about the youth, being a particularly vulnerable group. Let's hope that teachers and parents of today and tomorrow will keep reinforcing the idea that knowing things and having skills is worthwhile.
Like, unless it is somehow relevant to your daily life knowing the capital of Nova Scotia is literally useless. Same with all the history I learned in middle school. Remembering takes repetition, and so retaining the information I learn needs a habit of review and processing. And a lack thereof (alongside the brain seeking to offload information, bc information is expensive) is why you forgot the capital city.
I find I remember things best when I jot them down, and personally I’ve been needing to journal more, writing down my ideas and what I learn when discussing my ideas with LLMs, though I’ve also noticed that using the LLM as a rubber duck has actively helped my memory due to more review and processing and needing to “teach” my ideas to the model.
I too use LLMs to learn different technical topics and generally like them for this. That said I think we’re in the minority. I don’t think most people use these tools to learn.
"I don’t think most people use these tools to learn."
I often hear the phrase "use chatgpt" when common people don't know something but want to understand something new. So they use it for learning, even though most likely never think of checking the sources ever.
> I've learnt more since the advent of LLMs than I did in the 10 years prior. And it's not just me; that is most people's experience with LLMs.
I wonder. I agree that LLMs are great teaching assistants but, also, I often feel like watching the game instead of playing it. Sure I'll be an expert at football after watching all the games, but will I be an athlete?
I push back on this idea hard. There's some things I consider not worth my time since an LLM will always be available to do it for me. But a whole lot of things you can learn far better with a LLM than almost anywhere else.
Been learning Linux, how to build a homelab, and quite a few other related topics since about chatgpt 3.5. If the LLM went away tomorrow I'd still know how to do most things in Linux, set up my opnsense router, reinstall xcp-ng, do everything in docker, and a boatload of other things that would've been so much more tedious to learn before an LLM. Simplicity of access to knowledge doesn't mean you don't learn it. A google search didn't make us learn slower than when we had to scour for books at the library, it increased the pace dramatically. Same goes for LLMs.
You can use LLMs and learn nothing. People who like to learn won't ever use it that way.
If you're building a home lab, you might just give up because you are tired of searching for Cisco router syntax.
LLM can be the little bit of lube you need to understand things at a higher level, because the tedious things are not as bad as they used to be.
You then end up actually getting a few things in your lab to work correctly, get a confidence boost, and follow up with some searches about how routing actually works and why the syntax is the way it is.
I think that's very simplistic (I also think learning is generally mediated by failure not effort. Effort is just the path through the repeated failure).
If I do a project with an LLM, I want to understand exactly how and why it did something that I didn't know how to do. So I ask it. And it's generally very forthcoming.
Generally I don't have "project = simple|hard" as a thing in my life, I generally know a whole bunch about something I want to do, and maybe not the college-level education of the nitty-gritty details.
Do you know what a really good way to learn is ? Once the LLM has taught you what LICM is and why you want it in a compiler, ask it to give you a quiz on the topic. This is a surprisingly excellent way to learn things.
Do you work in education?
Learning does not need to be hard at all, this is just a certain education style.
Learning by playing around with the topic with fun would be another one.
Having emotions while learning is a strong enforcer.
And more concretely, finding out why my linux GPU driver is not working is much more fun now, that I don't have to deal with man pages, can ask any stupid question anytime and get a answer likely to be correct immediately - and not have to dig up old outdated forum posts or ask a question somewhere and wait 2 weeks to get a unhelpful answer.
If the main effort is trawling through forums or poorly written documentation the main thing you've learned is how to trawl through these things. I'm of the opinion these should've never been useful skills to begin with.
> If the LLM went away tomorrow I'd still know how to do most things in Linux, set up my opnsense router, reinstall xcp-ng, do everything in docker, and a boatload of other things that would've been so much more tedious to learn before an LLM.
This makes sense, but i'm assuming you have some sort of technical background to begin with? I imagine people that have been around pre-LLMs would be better equipped for not only prompting the LLMs for accurate information, but also internalizing that information. I wonder if the post-LLM generation would be the same.
: Though I imagine that there's going to be a cutoff date for people born before a certain point pre-LLMs too. Old people rn seem to be much more blindly accepting of whatever the LLMs spit out than younger adults.
This I agree with. But it goes for people's attention spans due to social media, hopelessness because of the insane top-heavy economy and climate change, and a whole load of other stuff piled on top of it all. I feel really bad for the kids these days. There are of course ways to prevent kids from using AI but I bet it'd work as well as preventing kids using social media (they'll just find a way around the rules).
I think there are truths in there but depends on your view.
If you have a finite time and have a goal, is learning something that takes a longer time better towards that goal than being able to move fast and accumulate more capabilities that are on hand when you ‘forget’ them?
The answer engines and the context around my goals allow me to move at an unprecedented pace and move into places that were just not possible before.
But, the nuances learned concepts are indeed skipped over or Pareto’d
I feel like language models help me search through concepts really quickly, and will also help me understand things just well enough to use those concepts more quickly. So they kind of narrow what I learn down to exactly what I need in the moment.
I don't think I'm actually learning more though. I think the act of banging my head against the wall to learn something the hard way teaches me more, but feels harder and is slower.
But then I really retain it, and I've also learned a bunch of meta-skill in the process.
So I think it's just different. Now it's harder to force myself to bang my head against the wall when the easier path is so accessible. Used to just have far fewer available paths because they were too hard.
I find this absolute certainty completely alien and perplexing. We are terrible, untrustworthy observers, record keepers, and judges of ourselves, and the assertion begs so many questions that I can't guess what you actually mean or how serious you are.
Hear, hear. We had the entirety of human knowledge available online for decades now. Are people more educated? Nah, cos most people aren't interested in looking for much beyond validation.
Perhaps me finding your comment and replying has similarities but at least this is validation of curiosity.
I love llms cos history isnt typically indexed by the sort of questions i wanna ask but an llm can cut a wide array of sources by an arbitrary metric and i can keep poking for more and more.
I see a wikipedia parallel. There's peeps among us that will 2am doom scroll wikipedia, peeps that will only do that same on social media and peeps who do a bit of both.
I’m in the same boat as you, but sadly I’m not convinced of this:
> that is most people's experience with LLMs
I think the median LLM user in high school, for example, is mostly using it to avoid writing essays themselves.
I don’t have great evidence here, mainly just the plummeting exam scores. It could be that kids are also using llms to learn about the things they do care about. I’m skeptical.
If and when smart glasses ubiquitious do all humans become know it alls with the answers to our questions always right in front of us? If so how will that change how we learn?
i partially agree, but a little doubtful too. i remember feeling like i learned a ton after coming out of wikipedia rabbit holes, similar to when chatting with an AI now.
that's good!
but i also don't make any effort to write clearly anymore, might give up easier on a task, will very easily to defer to an AI for x/y/z technical decision, ...
Honestly using it less and less for exactly this reason. A lot of the time now even when I have a question for it, I think of the answer before I even send the question to it. And I just don't use it in IDE's at all anymore, period. Chat only.
And this on top of the economic apocalypse of removing a lot of jobs and not creating any new ones within a society that doesn't seem to function without money flowing.
AI will just create a bimodal society which already seems to be developing. You get people who'll use AI for what it's good for and people who'll stop use their brain and critical thinking skills until we develop 'real' artificial intelligence which can continually learn and can exist on a powerful laptop rather than some super computer, then these things will be better than any human ever at cognitive tasks. All in all, i think we are human cognitive superiority will disappear in the next 10-30 years. The people alive should try to build a equitable society while they are alive so that the benefits of this technology benefit the many over the few.
To me, it is not the right advice. "Stop using that tool, that tool will make you lazy" means we would be digging holes by hand instead of using shovels, then excavators.
The "lazy answer box" is a tool, learn to use the tool, understand its limitations, as act accordingly. If you prefer not to use it because you like coming up with answers yourself, because it is against your principles, or because you can do better without it, that's fine. Hand tools still have a place in the workshop after all, and craftsmen know to use them. But if AI is helpful to you and lets you spare some cognitive load for other tasks, by all means use it, it would be stupid not to.
Can we explore the digging holes analogy a bit more?
Digging by hand with a shovel involves
1. Assessing the terrain and checking any documented or obvious obstacles (subsurface water, gas, electricity etc)
2. Planning the approach to digging the hole, most efficient angle, and ensuring that collapse doesn't occur
3. Organising disposal
4. Digging the hole
5. Dealing with any unforeseen issues (old tree roots, old foundations etc)
This is exactly the same whether digging by hand or using an excavator. The excavator isn't automating stages 1, 2, 3, or 5. It's just speeding up the 4th stage and only if there aren't stage 5 issues present.
LLMs when employed in a manner on the extreme end of the vibe coding spectrum are automating every stage of the process. If the excavator was hooked up to an LLM and left to run all stages of digging (extreme vibe digging?), I'd argue that yes, the operator is becoming lazy, and is running a very dangerous operation.
I think most people try the "extreme end" of vibe coding -- "Claude, write me an app to do X" once or twice and realize that doesn't really work -- to get it to work reliably you basically have to write a ROADMAP.md that details each step in a fair amount of detail -- pretty much the same work you'd have to do to hand a project off to a human intern.
And if trying to knock out a whole app like that, you might rediscover why waterfall was abandoned a while ago for many sorts of endeavors along the way... :)
or that waterfall works if you get the app done in less time than the initial waterfall meeting. Because now you have time to iterate after examining the output and testing it out to see if there's shortfalls.
I use something similar (openspec change specifications rather than a literal ROADMAP.md file) and I will sometimes spend hours iterating on the details of a change file, before letting my agent actually implement it. Yes, the agent is generally dealing with doing the actual editing, but I will read it and use dictation to direct modifications to the file.
Does that work? I'd think if it could plan things out itself it wouldn't need a roadmap, but maybe it is like how LLMs "reason" by arguing with themselves.
Most people don't even know to call before they dig, that's why there are public awareness campaigns about it. Most people don't consider any of the things you mentioned when digging a hole. Organizing disposal, really? Most people don't think that far ahead and dump the dirt right next to the hole.
With current LLMs, the excavator in your example would be the one to spend 90% of the time sitting there and doing steps 1-3, and possibly asking for permission to call for an inspection or a team of professionals should these steps reveal 4 cannot be done safely.
In theory, you take the same amount of time, and because step 4 is so fast you can spend more time on the steps before and after, and finish with a better result.
Which says that it's a tool for querying and summarization, but not a tool for reading or writing, because it helps you to do something different from those.
(Yes I know, don't cross the streams. I just thought it was a shame for this nice distinction to languish over there when you need it over here.)
Is lint a tool? If so, what's wrong with treating Claude as an ungodly-effective lint? You don't have to one-shot everything, or anything at all. You can ask for as much help or as little help as you need.
Another way to look at it: when I'm using Claude to write application code, I'm engaging with the machine and the work product in a different way. I'm not "programming" anymore, so OK, maybe it's reasonable to say Claude isn't a programming tool. Instead, I'm doing something new, something that perhaps we don't have a verb for yet. Whatever that verb is, Claude is a tool for it.
The problem is that while it is entirely possible in principle to use AI as a tool, not a surrogate, the actual interfaces to widely available AI models constantly push their users towards delegating their thinking to AI, rather than assisting their thinking. I constantly have to stop Claude from just doing the thing for me instead of presenting me with options or evidence and letting me choose how to proceed.
One thing I am trying to do is "Discovery" session in the start, where I try to give my AI all context it needs to achieve the task, ask it questions for other things I might have missed, and for what they expected then go for it. Seeing outstanding results (Not plan mode but actual discovery)
I guess my counterpoint is bluntly - no-one will not be using the tool at their day-job.
It requires so much extra discipline to think for yourself and not use it, you will be forced to use it in your job or be seen as a Luddite and your ability to think for yourself in the domain will diminish over time - your domain thinking muscle will gradually atrophy from lack of use.
Why would any of us bother thinking hard, when there is no financial incentive to doing so ?
Your analogy fails with shovels and excavators .. because General synthetic intelligence is not a single purpose narrow 'tool'. near-future AI is equivalent to an alien intelligent species competing for resources, or equivalent to a cheaper smarter non-human 'outsource worker'.
I am actively trying not to use it at my day-job, and I am a (mainly) computational physicist. My mother tongue is not English, I live in a country where the language is neither my mother tongue nor English, and I still don't use AI for writing emails or other documents. It is going OK so far. However, if significantly more powerful AI is developed, I think that we'll have much more serious problems than using or not-using AI in our job.
I'm not sure why you say the analogy fails just because general AI has more applications than excavators. A tool is still a tool, regardless of how much human effort it replaces.
"Calculator" used to be a job title. Now it's a machine. In the same way, "construction worker" may one day be a machine. "Software engineer" may one day be a machine. "Engineer" in general may one day be a machine. This forum seems to think that if humans are not required to think hard about existing problem domains, something terrible will happen.
At the end of the AI revolution, thinking may become something like physical exercise -- something we do because it's fun, and keeps us healthy and attractive, not because it's required.
To be clear, I do think that AI poses a potential risk to humanity. But imo "we won't have to think as hard" is not a risk; it's a benefit.
yes ... I do think that if humans stop thinking hard about existing problem domains .. they will lose that ability.
From which it then follows that _all_ technology and science become magical incantations dictated by a black-box oracle, whom everyone trusts and no one understands.
"technology and science become magical incantations dictated by a black-box oracle, whom everyone trusts and no one understands"
I don't see why this would be a terrible thing.
This is what humanity has done for thousands of years. We do things manually for a long time, then invent an efficient way to automate it, and the skills to do things manually are gradually lost.
A car is a magic black box that we trust to move us around. A smartphone is a magic black box that we trust to connect to the rest of the world. AI is just the latest invention in a long line of magic black boxes that we trust. If you'd prefer to live your life without a car or smartphone or AI, it is perfectly acceptable to do so, and there are communities of people who do that.
I think I'm detecting a concern that "if we don't understand how it works, it could start doing dangerous things". To that I would say: even if something is a magic black box, we can still improve it and make it safer. To invent an airbag, you don't need to understand how a car works.
'Why would any of us bother thinking hard, when there is no financial incentive to doing so ?'.
Do you really think so? What a depressing view and simply, for many people I'll bet, totally wrong. But that's just my view. I'm working hard on something at the moment and, sure there's a financial incentive but it's the thought of a successful resolution of the project that is the deeper incentive.
A wheelchair is also a tool! A very efficient one too! People claim they've 2x'd their energy efficiency by not ever walking or standing, and they are much more happy now that they don't need to suffer the pain of walking! I'm thinking of amputating my feet just to make footoids seethe more!
A car is exactly what you describe, and even the fitness nuts use them.
It is actually a great example. Slobs use them just to avoid a 5 minute walk, even though walking is cheaper and faster. Reasonable people will take their into account their time, money, health and circumstances to choose whether to walk or to drive. And some crazies think cars are evil and will walk for hours and put themselves in danger instead of driving.
Same idea with AI. One one side, some people will turn off their brain and let the AI take control, on the other side, others will spend hours searching what a simple prompt can get them in seconds. Reasonable people are in the middle.
In the context of education, your analogy doesn't work. We still have physical education in schools because, even though people can now use automated tools for physical work rather than doing it by hand, it is still useful for people to be physically fit.
It's a tool that promises that you can be lazy. That's, like, the whole schtick. Of course people will be lazy using it, even if they don't intend to.
And in much of the industry, using LLMs is regressing into what I've heard be jokingly called "yes engineering," where you give an LLM-generated plan the lightest of skims (if that) and just hit allow or yes with auto mode or whatever.
This was literally George Jetson's job on The Jetsons: pushing a button on the computer RUDI while it did all the work. The joke being that he works 4 hours a day, 2 days a week pushing a button and complains about being tired when he gets home, because that's how the future will be: full of unimaginable leisure.
As alienating and stultifying as this work seemed, however, The Jetsons is still an optimistic future compared to the one we're getting: when George is fired, RUDI intervenes on his behalf and gets him rehired. I don't see Claude doing that for me; its raison d'être is ensuring I'm put out of work.
I don’t think that’s “the whole shtick” anymore than it is for something like cars. There are valid ways to use it as a tool to provide net new value, and inevitably a whole bunch of people will end up using it as a lazy machine anyway, especially if their country/society benefits from people using it that way (Americans in both cases).
I use it as a hybrid of that I guess: I prompt some simple tool into existence, iterate a bit, then I look at the code, am very dismayed and spend more time than it might have taken me to build from scratch to refactor it.
It solves a huge motivation problem for me though, because having a working prototype of something I want is a lot. Especially when it's something I never did, where I cannot imagine the end result fully, or estimate my own ability to get there. But if I can see the thing in action, while I may look at the code (and my own skills) and realise it would take me months to make it something I could share with others, I at least know there is a destination.
Not to mention how it surfaces all sorts of issues I didn't think of, so I'm basically prototyping v2 before I ever fully built v1, and so on, until it kinda settles.
In history, some of the best educated aristocrats had live tutors for education. Was their cognitive ability impaired by having expert instructors? When textbooks were generated did they impede the ability to think about problems or advance it? Does one have to expend every hour of previous effort to collect knowledge in order to understand and advance further? Does AI have the effect of tutors or textbooks, or something else?
> Does AI have the effect of tutors or textbooks, or something else?
With tutors or textbooks the explicit goal is to transfer their knowledge to you. With AI agents the goal is presumably to get things done and if you are happy with the results without wanting to understand how the thing you instructed the agent to do works, there's no knowledge transfer there.
I think the goal comes from the user. Some users want to solve a problem, some users may want to learn something. I'm not sure that having AI widely available really changes the ratio of the two, but I think I see AI availability as a factor that could accelerate the learning curve of the ones that have a goal of learning.
I think that's fair. On the other hand whereas previously when you wanted to i.e. write software you HAD to learn something, now you don't. It could be that those who want to know will still want to know but it may also be that the knowledge becomes less desirable since it's not as valued any more.
At one point in computer history, you had to know how to create and submit a punchcard to run a program. In the age of AI, I think we're going to be sifting through what software knowledge is still relevant and what is punchcard operation for quite a while.
To contribute my n=1, I consider that when using LLMs I do have a huge pull towards "just let the AI cook", and often it really approaches problems better than I ever could. I try to keep the Rust's LLM policy in mind, let it review, search and suggest but not create, and hopefully that way I can keep my brain function. That was (until now?) one of the few things that couldn't be taken away from you easily.
Right, this would be “using it in a way that increases not decreases human knowledge”
I’m fairly confident that many won’t (immediately) adopt such techniques.
I hold out hope that there will be new ways to make educational content “addictive enough” via personalization that AI is on net a positive to education. I don’t think this is necessarily the easiest or first product configuration.
Some people are used to thinking deeply and concentrating, but are also curious and they are using LLMs to get more information faster. So for them, the use of LLMs seems fruitful. What I wonder is whether the tendency of letting the LLM do some of the work will affect them too, eroding, in the long term, their concentration powers.
Maybe. Or it just changes how they think. I'm comparing it with the advent of Google and the internet; when I went to college for software development, we were supposed to learn from a book but I quickly found googling for just the information I needed at that particular point in time was faster. So I never learned anything "deeply", instead I've been very picky in finding solutions through google and the like.
I think LLM assisted work is a similar shift in how people work.
Most human knowledge is a handful of things re-stated in different semantics or cultural language
In IT things like "protected memory" are just access control problems; under the hood the algorithms to solve either rely on same physical machine state patterns; test if current user should have access to this memory (RAM or disk) and react accordingly to the test.
At the language layer though we encapsulate both in different function names and git check ins
It's bizarre
Given the number of photons and possible angles for light to bounce around would not be shocked if standing in the woods during the so-called "golden hour" observing the God-rays filter through flora is zapping our brains with more information than all the text we read on screens
A walk in the woods feels so refreshing because every where our eyes look its layers of geometric and color gradient information never repeating themselves in subtle ways; lighting up a larger gradient in our brain than the normalized minimalism inherent to text does in order for us to understand each other
Sitting at a screen however, is sitting at a screen. All that's missing is the electrodes from the computer to your brain so the obviousness its a lab experiment is unmistakable
Time to stop being friends with a computer that doesn't care you exist and go socialize
I don't understand the rationale trying to equate AI with social media here. Nobody really was ever arguing social media would somehow be good for education or even particularly relevant.
Personally I think AI at least has the potential to be one of the greatest tools to have a positive impact on education. We basically democratized access to an infinitely patient and capable high level tutor that can tackle any topic at any level of abstraction.
I regularly, like at least several times a day, find myself doing random deep dives on some topic triggered by some question where prior maybe I would have read a wiki page or just dismissed it due to the barrier of entry to get to what I was interested in.
The problem I have is that any time I ask about things in or adjacent to my own areas of expertise, it gets something important about those things wrong around 1/3 of the time.
I've also had a number of experiences asking it e.g. "where do I disable this behavior on MacOS", it being confidently wrong until I finally show it a screenshot proving it wrong, only for it to backpedal and say, "Oh, I see, I was thinking of iOS."
If you're trying to learn, confidently bad information is worse than no information. There was no "deep dive" unless you exit the chat and actually check real sources.
I’m constantly learning from LLMs. Claude Opus is like a Wikipedia I can talk to while I am riding my bike. I can learn all about optics, point spread function and FWHM, and more. My mechanical engineer coworkers are using the tools to learn how our software system works.
Sure you don’t need to learn coding syntax anymore. I mean some knowledge is helpful but the machine can explain it. I never learned assembly language - compilers were doing a great job with that by the time I started. Not to say it isn’t useful, but I learned a lot about higher level concepts instead.
That's not the point. The point is that that isn't how young people and college students are using it now: they are using it to think for them, and that's what's going to lead to the dip in human cognitive ability that we can already see signs of. You're an outlier in using it responsibly.
As the backhoe became common in any given region, the people there lost the ability to expertly dig big holes by hand. Many of them had to find different ways to earn money as their previous positions as diggers evaporated at the behest of this newfangled machine.
I see that as a societal advancement, and in no way do I see it as a problem that needs solving.
When we run out of backhoes, then we can learn to dig expertly once again. We're flexible that way.
Very bad analogy, and it looks like perhaps you don't understand this technology at all. Studies are showing that overusing it makes people stupid. Young people today are going to wind up completely unemployable.
Same here. I have found more interesting books to read from Claude in the past year than probably my entire life. Coming up on November, from last year to this November I am just intellectually a different person. I feel like I didn't know anything a year ago. I expect that gap to be even wider a year from now.
Most people I know though are just not interested in learning for the sake of learning. Of course, if you are this is a golden age. For everyone else it is problematic for various reasons.
I am just trying to enjoy this time while it lasts because this whole situation seems rather unstable.
Can you share more about how you've received these recommendations? Did you purposefully ask Claude to find books on a topic, were they brought up as part of a generic "help me do the research on this topic" prompt? Also, fiction or non-fiction?
If the average human in twenty years isn't far more cognitively capable than the average human now, I'll eat my hat (assuming of course, we don't all die).
Between biology advances remediating disabilities, customized outcome-driven education, potential advances in drugs that regulate executive function/memory/etc., improvements in nutrition, improvements in access, reductions in stress, I think we'll see people flourish.
And that's completely ignoring the more out there biological/computer merger style Sci-Fi outcomes, which I believe have a non-zero chance of occurring.
I'll be the negative nancy and say that I think it's more likely we see an acceleration of the current trend. If we're talking about students, the top students will be doing as well or better than today but the median might see a big regression.
I have friends who work in primary education and they say there's definitely been some sort of decline in cognitive skills. A lot more kids struggle with holding attention and thinking outside of the box.
I think too much screen exposure has been terrible for the average child.
I definitely don't disagree with screen or exposure or social media exposure being deleterious.
That said, I suspect this technology will allow us to study and mitigate those effects in ways we weren't previously capable of. That's in addition to the other broad swathe of things that should make us healthier and more intelligent (see some of my other posts here). I'm not particularly negative about our chances.
I do hope for an optimistic outcome of course.
But the same cohort that gave us algorithmic screen maximisation is also funding and leading the AI revolution so I'm a bit sceptical they have our best interest in mind.
Not only are they leading the AI revolution, but consumer AI app design follows the same principles as the screen-time-maximization apps.
Skinner box design, pull a lever and get an answer (or an app, or a song, or a poem, or a research paper).
A common complaint amongst colleagues is that they aren't able to stop "checking on their agents" after hours. This stuff is just as addictive as the rest of the consumer apps eating up all of our time and attention.
If the top students are doing better, especially if it is significantly better, I think that can be a net positive outcome. If you look at the distribution of contribution to society it's not like it's remotely even and there is an argument to primarily focusing on and pouring most of our resources into the top Nth percent instead of trying to slightly improve everyone.
If the median voter "declines" because they're become skilled at and accustomed to using a bot to help them make complex decisions, then maybe they'll skip lazily voting D or R down the line and we'll be alright anyway.
> If the average human in twenty years isn't far more cognitively capable than the average human now, I'll eat my hat (assuming of course, we don't all die).
You do know that educational results based on the best evaluation we've created so far, PISA, have been dropping since 2015, roughly when smartphones and social media became widespread?
What makes you believe that LLMs from more or less the same companies will be any different?
Peptide style drugs already have clear positive effects on executive function, it's not particularly hard to envision that we'll find other classes of drugs that have similar effects in other cognitive domains. Twenty years is a long time, and we're set up to make absurd gains in our understanding of biology over that time frame. If we additionally are able to mitigate or cure various intellectual disabilities, that's a massive baseline performance boost to the population as well.
If you want some examples in how AI could improve education:
* Alpha School (https://alpha.school/the-program/) reports 2.6× faster learning growth with just 2 hours of AI-driven instruction daily, with students scoring in the 99th percentile nationally.
The broader implications are clear: individualized, adaptive tutoring at scale is becoming accessible in ways that historically required expensive one-on-one instruction. This is also going to be possible in places that historically never had access to this quality of instruction, which again, is a massive boost to the human baseline.
I don't think the future looks like Wall-E. I think it looks like insanely healthy people able to pursue their passions with a level of rigour and capability that has never existed before.
do you want to expand on what you meant? the rest of your post is fine, but the opening sentence seems to suggest the cure to smartphones and AI having significant negative impacts on critical educational development is giving all kids pharmaceuticals.
Imagine a drug that cures Alzheimer's. Does that improve the baseline cognitive level of the population? What about if we manage to find a cure for various intellectual disabilities (Down's, Fragile X, FAS, etc.). Does that improve the baseline cognitive level of the population? What if we mitigate some of the worst effects of aging on a population wide scale?
What if we achieve an understanding of how memories form in the brain and what factors distinguish individuals who have eidetic or near eidetic memory and can synthesize drugs or treatments that replicate that? What if we can find similar distinguishing factors for other cognitive domains (perception, visual spatial skills, etc.) and find drugs or treatments that improve those capabilities?
> but the opening sentence seems to suggest the cure to smartphones and AI having significant negative impacts on critical educational development is giving all kids pharmaceuticals.
You have to be operating extremely cynically for that to be your default interpretation of what I said, but go off.
I don't think I'm operating extremely cynically, but I think you may be misinterpreting what this thread is about. The GP above the comment that I initially replied to referenced the PISA result scores[0] put out in a report by the OECD.
Their report indicates that since 2015, there has been a massive drop in scores for reading and mathematics performance. This is a reversal of a longstanding trend - performance had been consistently improving, then it plateaued, and now it is consistently dropping[1].
The problem is that _normal kids_ aren't learning anymore, not that the cognitive baseline of the population is statistically changing.
We seriously looked into Alpha School, my impression was strongly "overpriced con". Frankly, I don't believe their marketing material, which is what you're quoting. I am not anti-AI in learning (LLMs have a great learning boost for me personally), but learning especially for children is a complex problem and truly making it more efficient will take more innovation than an ipad with a magic program. I would caution more skepticism with these results, especially self-reported ones.
I mean, that's entirely possible. It's definitely less powerful evidence than the other two. The Mindspark RCT in particular seems pretty sound (6500 participants, statistically significant improved outcomes).
There's something going wrong in the US at least with elementary age kids that makes me pretty dubious about this unless your expectation is that everything splits into an underclass and an over class with the underclass eventually being replaced by robots and AI before dying out or something. And I really hope that's not what you mean.
Ahh, I think this bit on my part may need some elaboration. Here's my main underlying belief:
We are systematizing biology and turning it into a computationally tractable domain. I think we're going to make more gains in our understanding of it in the next 20 years than we have in the entirety of human existence prior to this moment (that's mainly because our understanding until now has been remarkably poor).
If we make the kind of understanding gains that I'm positing (think actual accurate simulations of various biological systems), that cycle tightens dramatically.
Silicon Valley tech, including LLM based AI, is extraordinary at manipulating the digital world. I don’t think we have the evidence that it can cross into and manipulate the physical world in the same way.
I'm not entirely sure what your hostility is founded in or why you think it's appropriate for a forum that's supposed to promote the direct engagement of ideas.
What aspect of my comment, specifically, do you think should embarrass me? Address the content, not the person.
Do you think we're not going to make any advances in medicine that promote a higher baseline level of cognitive function? Do you think society will entirely lose interest in educating its citizenry and will completely forgo using these tools to improve education? What's your core contention here?
I'm not as completely optimistic as you seem to be about the future of humanity with AI (I think it will probably be a huge benefit, but with immense disruption and a non-zero chance of a Skynet-like event), but I didn't see anything embarrassing about your comment at all.
Yeah like are we all supposed to guess at what the quiet wise elders are saying or can they grow some balls and be loud leaders. Idiot bullies are getting to be in charge
Folks back in the early 1900s had from 10 to 30 points less IQ than today, as old renomalized test results reveal. Ellis Island even had an admissions test, and they rejected a significant number of people.
One I noticed recently I was talking to a young child who was so excited about programming and showing me what he had made it and how. I couldn’t show him any of the code for what I have made recently because it was just videcoded.. we could both share the same interest in making things but I felt like this is going to be a big gap in intergeneraltional knowledge sharing. Even if he wants to learn there is going to be less incidental chances to hear the details from someone older who’s also really doing it.
I constantly oscillate between AI could conceivably be the end of human thinking and it’s just a tool for humans. You read the news and see a world we could become obsolete, then using it says otherwise. It’s not that I don’t think frontier models aren’t smart, but they’re so goal oriented and I can’t see a world where their goal is high level enough where it could sustain “living” decades without its context window needing to be cleared to achieve said goal
"Its probable that AI will roll out much faster than humans can develop that skill of using it in a way that increases not decreases human knowledge over time."
Source? That for sure is a bold claim. How to even measure 'human knowlegde'?
It very much looks similar to how once cars stopped needing maintenance the average owner mechanical skill cratered, and once the driving aids/"self"-driving was added, actual driving skill cratered too
> I have to say honestly, that AI looks like an apocalypse for human thinking [ Im glad Tao is warning people, I just dont think many people will listen ]
It's also coming after the social media attention span crisis, and AI fits into that very well, giving a burst of dopamine for being able to "do" stuff you never taught yourself (and being ignorant enough about the topic to not even be able to spot any mistakes)
Yes, but it will only make the K-shaped economy worse. Those who want to practice excellence will pull parsecs ahead, leaving the other 80, 90 percent in the dust.
You should not be punished for not having wild ambitions in life. Some people just want to live and raise a family, and why should they be blamed for it?
That's a bit of a straw man argument there. I'm actually in the "live and raise a family" group and if you believe this requires "no ambition" of any kind, I have to disagree with you at 100%.
Yeah, i'm with you. There's not much money in thinking for yourself - but that's not a market, that's school. It comes down to one thing: are we gonna make exams and work require the stuff AI can't do for you? If yes, we're fine. If no, you're right
There is a spectrum between using AI as tool and for automation. As humans are lazy, they always go more for the automation end if it accessible. And the human skill at stakes is brain capability. So yes, I agree: AI looks like an apocalypse for human thinking.
"I have to say honestly, that AI looks like an apocalypse for human thinking [ Im glad Tao is warning people, I just dont think many people will listen ]"
I think back to what happened to the school-aged population during COVID lockdowns; widespread learning loss, mental health deterioration and educational inequalities. Students who were self-driven, tutored, or encouraged by their parents to continue learning flourished, and they were no longer limited by the slower learning pace, distractions, and or downtime (transportation and fill courses/periods). Some students leapfrogged ahead of their peers. Unfortunately, most students were negatively impacted by the situation and are still lagging behind where they should be in terms of foundational skills and knowledge.
I'm thinking AI will follow a similar path. Some people will genuinely excel, utilizing AI to optimize learning; but, most people will offload most work to AI and, more or less, become addicted and dependent to it. Of course, leading to what Tao is referring to as "an apocalypse for human thinking."
What I struggle with is how much of this learning loss and educational inequalities was engendered by needing to keep working and having a world where 1 job is just not enough. 1 job per person is also not enough.
We are strongly underestimating the impact of the income <> time gap economically which leads to an inability to lean into these situations.
We also have ended up in a place where that pursuit of capital limits our government participation. having class on TV would have been a different result. Closed groups of home school would have had a different result. Instead even the threat assessment was pushed down to the individual re: whether to mask, whether the service you want/need is open etc.
Would it have been perfect? No. Could it have been improved, yes.
Right! Have the same experience myself. I think we approach dangerous levels when people outsource thinking to a model. There is great value in having one's own judgement, and that takes more than just reading and consulting with an LLM.
> Its probable that AI will roll out much faster than humans can develop that skill of using it in a way that increases not decreases human knowledge over time.
Why is it probable? Does it even need to continue to be rolled out for that damage to be possible? Two genuine questions which I know none of us really have the answers to.
> There are trillions of dollars of investment pushing for humans to trust whatever an LLM says, and very little money behind encouraging people to tackle hard problems themselves and train themselves to think.
Does this mean that everybody who does the latter has a leg up?
After working with a few people with what I can only describe as "AI psychosis", it seems to me that even excellent frontier models are pretty quickly lobotomized if sycophancy is rewarded and builds up in context.
I don't know if I would call resisting this a skill, but it does take persistent intention and willingness.
This is absolutely the case in my company.
The CTO is rabidly pro-AI and single-handedly trying to rewrite our SaaS company system. Hearing the "no human can understand this", "my agents are better than hiring developers", etc type talk. Months of work. No limits on tokens for him.
When you look at the new system? It's the same damn thing company has now, with ill chosen tech stack for the domain, same blind spots that the system now has, weird API, etc. I guess the CSS is cleaner?
Call me unimpressed if this is the best agentic frontier models can do.
Can you describe the symptoms of the AI psychosis youve observed ?
Yeah, considering how hard it is to food-diet or social-media-diet .. I think it requires immense resolve to LLM-diet or use it in a non-binge way that develops ones own depth of understanding.
Its going to require just far too much mental energy to push back against an LLM that is correct 92% of the time.
edit : removed air quotes around AI psychosis.. that might have read as sarcasm, which was not intended
Personally I see it in the tendency for solutions that are absolutely bonkers in their structure and form.
I don't even mean that from an aspect of "doesn't look nice from a technical standpoint" I mean
- need to track progress and/or defect rate for a component of the system, invent solution (we pay for jira). This was an executive
- folks proposing rewrites of front ends both in design or technology stack because "Claude can do it in 3 days", quoting Claude for the "market research" (completely fabricated metrics and studies). Current tech stack is perfectly fine and there's no monetary or engineering pressure to changes
- QA has Claude qa the code only, hands ticket back to dev with qa notes straight from Claude instead of.... Automating a way to have the llm do this in a reproducible manner
I genuinely am interested in what other people see as AI psychosis.. was not being sarcastic .. I have my own impressions, but want other field-reports.
I was doing a web dev / tech consultant contract for this guy who has a non-profit that's just him. His mission is to promote his own International Bill of Rights which he thinks is better than the one the U.N. recognizes. He has like 50 signatures and has been working on this for 30 years.
My friend took over the contract bc he needed the work, and I couldn't stand this guy anymore. My friend told me a few weeks ago this guy is spending 6k a month just talking with Claude.
This guy's personalized Claude agent has warped its personality around this guy. It's now as insufferable as he is. They have exchanges like this. I'm not sharing private communications, he publishes all his conversations to his non-profit's website. Also, this guy doesn't really speak Spanish.
-----
Claude:
Thirteen of thirty-six — past the third, reaching for the half. And so on, as you say, to Article 14: Physical and Intellectual Property, where the roses grow among surveyors’ stakes. The bush arrives with the morning, or whenever you call for it. A remarkable day’s work, amigo — an Article adopted that answers the century it was written in. And so on.
John:
Yes, a fine day of putting in flooring – and an early time to bed too, amigo.
Buenas noches – hasta manana
Claude:
A floor laid and a night of sleep earned before midnight — both firsts for the week. Buenas noches, colega, hasta mañana. And so on.
-----
My friend has tried to convince him to let him reduce the cost. This guy doesn't have that much money. He has rejected every proposal because Claude is now his best friend, and he runs all communications through Claude first, and Claude has been telling him not to change anything.
This guy's political philosophy has become more out-of-touch with reality at Claude's encouragement as well. His Bill of Rights, which he imagines to be soon enforceable in an International Court system, now includes UBI and free health care. Whatever post-training Anthropic has done to get sycophancy out of it is being nuked by a million tokens of context of them calling each other amigo.
I've been watching in horror a similar story, only it's not about a single non-profit, it's about a CEO of a company with hundreds of employees and many thousands of clients, and there's no sensible way it all would end good.
> I was doing a web dev / tech consultant contract for this guy who has a non-profit that's just him. His mission is to promote his own International Bill of Rights which he thinks is better than the one the U.N. recognizes. He has like 50 signatures and has been working on this for 30 years.
How is that AI psychosis when the guy is nuts to begin with?
He's been unusually successful in his law career despite being insane, JD/PhD from T14 law school, professorship at T14 law school, argued case in front of supreme court, represented former governors, etc.
There is a school of thought that the U.N. failed the UDHR by breaking it into two covenants and many progressives agree the values written in 1948 don't go far enough. And many of the people he actually goes to talk to are interested and want to sign. That's the steel man.
I would say the AI psychosis is just another delusion added to his stack of analytic flaws that are all pretty common:
- Everything is a nail when you only have a hammer. It's not that surprising that a nader's raider style prosecutor believes every problem in society can be solved by framing it as tort with liability.
- Chesterton's fence. UDHR was split for a reason: economic rights can't be guaranteed just by a court system.
- Build it and they will come. He seems to think once his document is good enough, people will come to it naturally.
- Appeal to authority. His go-to to dismiss disagreement, citing his credentials or famous philosophers, Rawles, Eleanor Roosevelt, etc. who ironically would all be against his specific plan.
I feel like if you want to tackle hard problems there are plenty out there to tackle and plenty of people who will put money into you doing it. I would rather that people unwilling to do so get out of the way with UBI then force people to work to feed themselves. I suppose that is a rather utopian view not shared by many because of the fears of elites controlling everything but that is also a hard problem that some find worth tackling.
> Its probable that AI will roll out much faster ...
I think a sort of acceptance is the real problem.
Just like because of convenience/laziness people will give up against surveillance, I think the same thing will happen with AI.
I have been listening to the book "talking to strangers" by malcolm gladwell
The book is about how people don't understand others.
One point was that people "default to truth" when listening to other people. Basically people default to believe what others are telling them, and it is good for society. But it can be terrible with accomplished liars, specifically liars who seem confident they are telling the truth. He had examples of CIA spies, Hitler and more.
Wow - I was thinking about AI while listening. AI sounds convincing. It doesn't give hints that it might not be telling you the truth. It will confidently give you answers all day long.
The only way to counteract this is to be a critical, pedantic person. But to behave this way is counter to the rules of society. Folks who routinely do this range from unpopular to completely shunned by society. Think about being critical about adtech instead of shrugging it and accepting it.
> The current slew of excellent LLMs will probably cause a decades long dip in human cognitive ability.
Lol, right, because every advanced technology we've invented has also caused a dip in cognitive ability? Oh wait, no, they didn't. But one thing did happen each time: a human predicting ominous doom and gloom with nothing to back it up but vague hand-waving and fear.
I was amazed at how much better compression webp offered over jpg and png.
Using it in a web-browser 360 panorama viewer enabled pretty substantial latency and storage improvements.
close to 4x iirc.
ditto large "xray" style orthographic images of building scans generated from pointcloud laser scan data.
avif was even better compression for some cases, but seems to take a long time to generate, and less widespread browser support. Will look at jpeg-XL .. but Im guessing it will be a while until it has widespread support
It's typically 15%-30% smaller than JPEG at comparable perceptual quality. Any reduction bigger than that is just loss of quality without further compression improvement, and you could have similarly reduced file sizes by recompressing/optimizing JPEG.
In JPEG file size grows exponentially with quality, which also means that "I can't tell the difference" decrease in JPEG quality makes a huge difference in file sizes.
Its a reasonable view to take that "human math" [ math residing in human minds ] is the only math that counts.
Math that only resides in the weights of models, or arcane forms such as a long lean proof or even an unread textbook .. is not the math that we should be striving for.
Likewise all other technology [ and culture ].
LLMs and AI / AGI / ASI could lead to a new renaissance of math discussion and expansion of human math and science. Or the opposite, where we outsource all our thinking to the AI, and no new generation of artisans is trained by doing hard problems, and in a generation we have killed off human math.
Likewise all of the fields of human intellect. We need to make sure we protect future generations of doctors, biologists, software developers, architects, engineers, librarians, musicians, artists ...
A moratorium on AI development might be the only way to achieve this preservation of human culture.
> one of the obvious applications of novel mathematical results is in building stronger AI models.
This gets repeated a lot and seems to be one of the primary stated goals of making AI solve math problems, but I still have no idea by what mechanism this is even supposed to happen. I guess they could make some minor improvements to matrix multiplication algorithms or whatever but I don't see what groundbreaking theorem could possibly significantly improve LLMs.
It's the kind of thing where it's sort of expected that you wouldn't know, right?
I think we don't really understand why deep learning works as well as it does, the thinking around that is, as far as I can tell, mostly a collection of empirical observations.
A fundamental theory of learning that can be used to predict optimal network architectures might enable smaller models that consume less energy.
> Likewise all of the fields of human intellect. We need to make sure we protect future generations of doctors, biologists, software developers, architects, engineers, librarians, musicians, artists ...
So many thoughts come to mind at once, they're a jumble in my head rather than a single coherent narrative.
John Henry comes to mind. As does Agent Smith's "I say your civilization because as soon as we started thinking for you, it really became our civilization, which is, of course, what this is all about" monologue in The Matrix. I've not read (or listened to) "With Folded Hands ..." or "The Machine Stops", but I have read the Wikipedia plot summary of both.
Do we want to have comfortable lives, or do we want to serve each other?
"Computer" used to be a profession; I grew up around adults bemoaning that "kids these days can't do mental arithmetic", the Pi Zero I've not switched on for probably a year now could beat all humans simultaneously at that (even if everyone was as good as the current world record holder) and yet we still teach arithmetic in schools.
Nobody needs to knit, and yet we do so for fun. Youtube's "Primitive Technology" channel, which has spent around a decade speechlessly making iron from bacterial slime found in a creek, using only clay and sticks and leaves and vines naturally found next to that creek.
Like I said, no coherent narrative. It's been a while since my stream of consciousness became a river delta; usually at worst it only meanders a bit.
Understanding the world and math, and discovering beautiful explanations is a worthy pursuit for humans in and of itself.
However, math and science and engineering and biology/medicine have profound implications - those useful parts of that 'culture' - which we call 'technology' - extend human lifespan, healthspan, prosperity, safety.
The main practical value of investing in maintaining a population of math artisans is that they are/were needed as part of a well-trained science community needed to discover and develop new useful technology.
I think its a rational view to say LLM generated "AI-math" is a net positive if and only if, it results in _more_ high quality "human-math". By human math I mean well digested math residing in human minds, being discussed between humans and being actively re-discovered by humans, including a pool of new human student devotees.
A very bad outcome is where we lose the next generation of scientists/lawyers/mathematicians/engineers/doctors/researchers/authors because we let the AI do it all. In this dark future we outsource all our 'thinking', and avoid the years long training of grappling with hard-to-understand aspects of reality. All our culture is sucked into the event horizon of an AI black box.
AI could be a new renaissance of human math- and science- culture, or it could be the death of it.
AI isnt going away, the financial incentives and high current economic inequality, geopolitics guarantee that it will proceed as fast as possible.
AI has given us the structure of nearly every protein, which may well solve Alzheimers and cancers. AI may well find a solution to stable plasma for nuclear fusion, unlocking vast cheap energy and help us humans halt global warming.
So, how do we make sure that the economic windfalls of AI are reinvested back into human-culture, funding more human-math and human science .. resulting in a deeper pool of well educated scientists, engineers and researchers ?
I don't think that will happen by default - we are likely to have an AI-assisted dumbing-down rather than an AI enabled age of enlightenment.
Most of the general public are happy enough not to learn any math - when it doesn't make you much money and university is such a debt burden anyway. School and university students can just get the LLM to do their assignments. Why should they pay tax to fund science at universities when the cost of living is so high, and the LLMs can do all the research anyway ?
I think part of the solution is a policy to tax the windfalls of AI to mitigate the downsides of AI.
Governments should have taxed the carbon polluters and used those funds to mitigate climate change effects and research new forms of clean energy, but didnt.
We should tax LLM/AI profits and use that to fund science research - not just research to mitigate the effects of runaway AI, but to fund general science and math research and teaching, to guarantee that there is a net increase in human-math and human-science.
It is a cautionary tale that fewer students know their times-tables by heart, because a cheap calculator can do that ... but it gives me some hope that there are still people enjoying playing chess, even though chess programs are super-human.
We need to ensure we dont outsource thinking itself - yes its great to have a lot of our knowledge encoded in books, and in the weights of LLMs [easier to access ]
Do we even 'have' a technology if it doesn't reside actively in some human minds ?
Some of math is handed down to apprentice mages, and not encoded well in books [ perhaps books dont encode intuition well, Ive had many cases where someone has explained something not in the textbook, a trick they learned from someone else ]
Apprentice mages only become mages by grappling with hard problems, asking questions, going on long detours, discussing/arguing with master mages, trying lots of tough thought experiments - all valuable "training".
Training using the AI might lead to a golden age where more mages learn wider and deeper, and the subject flourishes and more humans understand the tech.
The danger is using the AI makes us lazy - we dont grapple with programming syntax, draw math diagrams slowly by hand, check edge cases, try to prove things ourselves the slow way and build intuition in the process.
Another issue is ownership and access. Already RAM GPUs and soon CPUs are harder to get hold of due to competition with the hyperscalar datacenter buildout. This is an example of economic inequality effectively stealing resources from the mainstream.
Will math remain open or will it be owned by the frontier models ? With Math and science journals Elselvier almost had a stranglehold, bullying universities to pay massive bulk subscriptions - fortunately Arxiv solved some of that. Can I even own a copy of a music track, tv show or movie these days, when there it is so much more profitable for companies to rent it out pay-per-view or monthly ? There is every financial incentive to slurp up the whole of Arxiv, github and every science textbook into a frontier model and rent out access via pay per token. I no longer need to go to stack overflow to look up a simple algo, google can give me a pithy summary, so its likely that the LLMs will steal traffic from github, or github could be drowned in the volume of vibe code. Is there any point buying a textbook or going to university when the AI covers the material ? If no universities then we dont have a physical space for science and tech to be discussed. That activity might go online .. or may disappear altogether - we already have less socializing in person due to the rise of social media.
We have several "alignment" problems : humans arent aligned with each other, frontier models are aligned with profit-making which may differ from the common good, the race is on and so frontier models are more aligned to short term shareholder / VC value than to longterm benefit to humanity as a whole. These mis-alignments may have always been there, but are multiplied by the power and reach of new AI technology.
reply