How does it say anything about the stochastic parrot theory of LLMs?
The only thing there that seems like it's relevant is the finding that the models they used had worse results when prompted in Chinese than when prompted in English. But ...
I am fairly sure I would do substantially worse work if I had to do it in French rather than in English, even if the work itself was all mathematics and programming and the like. Maybe that in some sense indicates that I am a stochastic parrot but it clearly doesn't indicate that I'm a stochastic parrot in some way worse than human beings are since I happen to be a human being myself.
... so what am I missing, that makes the models' worse performance in Chinese an indication that they're stochastic parrots in any interesting sense?
(I take it that "they're stochastic parrots just like we are" is not a very interesting sense. I mean, actually it would be quite interesting to understand better how much of human thinking can be reasonably described as stochastic-parroting -- fairly clearly the amount isn't zero -- but I think it would be interesting as a finding in human psychology, not as a fact about LLMs.)
I am using stochastic parrot in the sense that since models are mainly trained on codebases of Latin characters, their coding strength reflects on the "goodness" of the codebases that were fed to them. More Latin codebases, better English coding performance. I am assuming from the paper, that when prompting in Chinese, the generated code is in Chinese for as much as possible (imported libs etc.).
Even if we didn't know french, it is straightforward to use deterministic tools (I.e. a dictionary) to parse the instruction, writing out the code in English, then refactor as many words as possible to french. The paper did not give any indication that this was happening (would be a novel result indeed) and so I would have to conclude that it is running like a stochastic parrot, more English codebase, better performance in English only.
I think this might still be in the realm of "maybe about as stochastic-parrot-y as human beings are"; I can easily imagine doing a worse job of remembering relevant stuff that happened to be in English if I had to do my work in some other language. Human memory is surprisingly context-dependent. But it would be interesting to see what happens if you ask an LLM to write code while talking to it in some language that has waaaaay less programming-related stuff on the internet. Swahili, perhaps. Is it much worse than when you prompt it in English, or a little worse, or what?
If the LLMs are very stochastic-parrot-y -- just piecing together bits of code associated with the words you wrote in English-or-Chinese-or-Swahili -- then I would expect them to get catastrophically worse when prompted in a language in which there's very little programming content on the internet. (For what it's worth, I think I also think this degree of stochastic-parrot-ness seems rather incompatible with what they are able to do. But others may disagree.) On the other hand, if they're more like humans -- somewhat better at remembering relevant things when they're in the same language as they're working in, etc., but operating at a conceptual level as well as pushing words around -- then I would expect the loss to be much more moderate.
(It seems somewhat relevant that the insides of a transformer network operate on embedding vectors rather than literal tokens; presumably those embedding vectors are much less language-specific.)
I'm not sure I understand what work "on principle" is doing in this sentence.
What's the difference between voting against anyone who acts against your interests "on principle", and voting against anyone who acts against your interests "out of self-interest"?
(I am not suggesting that you shouldn't vote self-interestedly; arguably democracy works best when everyone does.)
Assuming I understand your question, I vote for people I would not have otherwise voted for merely to spite the people who have harmed me in some way. It's petty but it makes me feel like I have reclaimed some agency against said aggressors.
Candidate A - will do X, Y, Z all in ways I think are the best
Candidate B - will do A, B, C which I prefer less than X, Y, Z but in the grand scheme likely impact my day to day life little.
Candidate A - also supports (or is supported by) people / movements that actively make my life harder and piss me off.
That's not cutting off my nose that's deciding I can live with a less optimal solution to some problems to avoid something that pisses me off massively from growing more powerful
They told it to attack X (a simulated host inside their sandbox) and it attacked Y (Hugging Face, an actual external company). These are not the same thing.
We know what calculations it does. We have some hazy idea of some bits of how those calculations lead to something that at least somewhat resembles intelligent behaviour. But that's a far cry from actually knowing how it works.
For instance, suppose you give one of today's frontier models some of those chain-of-cubes rotation puzzles (the sort that infamously men are about 1sd better at than women, statistically speaking). How well will it do? I have absolutely no idea and I'm quite sure that a more detailed understanding of the transformer architecture would not make my guesses any better. (Actually, I do kinda have some guesses but they're based on a vague notion about how the models might be partitioned between vision-y bits and language-y bits, and it's very possible that that notion is out of date.)
> it does exactly what we expect it to do
Were you, let's say 6 months ago, expecting it to resolve one of the Millennium Prize problems?
(I do agree that it is more productive to ask "what can and can't they do?" than "should we classify that as intelligent or not?".)
> for Navier-Stokes they spent in 3 days more money than the whole mathematical community over the last 20 years easily.
Are you sure?
(The numbers I've heard, which I admittedly have no very strong reason to trust, don't seem that way to me.)
> Were you, let's say 6 months ago, expecting it to resolve one of the Millennium Prize problems?
I didn't expect them to throw millions of dollars at each famous math problem. But one year ago we already had LLMs that solved IMO problems, no?
> Are you sure? (The numbers I've heard, which I admittedly have no very strong reason to trust, don't seem that way to me.)
Math has very little founding compared to other science domains. Also, if you filter mathematicians by specialization in PDE and that have worked on Navier-Stokes, then you end up with a very niche community.
> For instance, suppose you give one of today's frontier models some of those chain-of-cubes rotation puzzles. How well will it do?
I feel like this is not the correct way of thinking about it. We can also ask, for instance, how well a state-of-the-art algorithm for the salesman problem works on a particular graph topology. People do PhD thesis on topics like that, so the answer is not obvious at all. For LLMs we still don't have a curated theory that explains what they're good/bad at, and that you don't see how to extract an answer from the definitions is no surprise since this is obviously not an easy problem. But all this is normal because this is a rather new topic (models of this scale appeared when? 3 years ago? That's nothing for science).
Anthropomorphizing LLMs has added so much noise to this discussion.
Yes, one year ago we had LLM-based AI systems solving some IMO problems. My impression is that most observers at that time didn't expect them to be solving Millennium Prize problems within a year.
> Math has very little funding compared to other science domains.
True. But to whatever extent the numbers I've seen are correct, for the whole mathematical community to have spent less on Navier-Stokes than OpenAI did -- even if we value the tokens they spent at something like market rate rather than at what the compute actually costs them (which might be right since any capacity they use internally can't be sold to customers) -- the average number of mathematicians working on Navier-Stokes since 2000 would need to be somewhere around four (depending of course on how well paid they are), and that seems too low to me.
> I feel like this is not the correct way of thinking about it.
It seems to me that if you say "It is absurd to waste time discussing whether it is intelligent or not. It is just an algorithm, we know how it works, and it does exactly what we expect it to do." then this only makes any sense if your "knowing how it works" and "what we expect it to do" enable you to predict what it can and can't do.
(I repeat that I agree that what matters is what it can do, not whether we choose to apply the term "intelligent" to it. But unless I misunderstood you were saying somewhat more than that.)
> Anthropomorphizing LLMs has added so much noise to this discussion.
I think sometimes it helps, sometimes it hurts, and sometimes it's indifferent, because LLMs are like us in some ways and unlike us in some ways. (The same goes for many other things, but LLMs are much more like us in some important ways than any other human-made artefacts.)
> How many human-years were invested in solving the same problem, before they were overtaken in the last few days by an AI? 90? Not even: that's just the time since Jeal Leray's statement of the problem in 1934. 26, if you want to count the time since 2000 when the Clay Institute named it as one of its Millennium Prize problems.
I hope this isn't actually news to you, but: There is more than one human. There is even more than one mathematician.
If there happen to have been as many as four humans working on Navier-Stokes at any given time since the year 2000, then that's more human-years applied to the problem than agent-years.
> How many mathematicians have worked on the problem? 10k? Not likely.
You don't get to count the factor of 10k once when working out how many agent-years OpenAI gave to the problem and again when demanding that for parity there would need to have been 10k mathematicians on it.
> And all that's without even considering whether the AI based its proof on carelessly shared work by the humans. Or rather, yes, let's consider that: it totally did.
Let's suppose that indeed what Buckmaster and Alpöge had done was in the model's training data. Well, it didn't enable Buckmaster and Alpöge to solve the problem for Navier-Stokes (they could only do Euler), and it did enable OpenAI's model to do that.
Also: we don't actually know that what they'd done was in the training data; the latest bits of what they'd done that could plausibly have been in the training data were from before when Buckmaster said they progressed from preliminaries ("We worked through the literature and upgraded various preliminary results") to actually making substantial progress on the problem ("This was until about a month ago, when we had real progress"); and from what Buckmaster wrote it sure seems like a lot of the Buckmaster/Alpöge progress was in fact done by LLMs. (E.g., Buckmaster says that he and Alpöge have been working frantically to try to understand the proof for their Euler solution. That sounds to me much more like "an LLM did this thing" than "we figured out all the hard bits and the LLM did nothing more than filling in a few details".)
Buckmaster's own account of things is that all the really clever ideas were those of Córdoba and Martínez-Zoroa. (Which are already out there in the open literature, and there is nothing remotely improper about making use of them.) And my understanding (but, note, I am not an expert on fluid dynamics or PDEs and I could be wrong) is that actually the OpenAI model's construction is quite different from that of C&MZ. On what basis are you confident that "the AI based its proof on" what B&A did?
(For the avoidance of doubt: I am not arguing that what OpenAI did was OK. Even if they actually didn't train at all on any of the Buckmaster/Alpöge chats, it's very much not good professional ethics to hear that someone else is working on something and rush to try to scoop them, and there is absolutely no question that they did that. The question here is how impressed we should be by the model's mathematical prowess.)
> A success rate of 17%?
A success rate of 17% on problems of this difficulty and significance is something that for any human being would be a career-defining triumph.
> We have thousands of problems solved by humans in the last 2k years and how many by AI? A couple dozen? Oooh scary!
That would be a more convincing argument if the AIs, like the humans, had been around and trying to solve those problems for the last 2k years. However, as you might have noticed, the state of the art in AI was rather primitive 2000 years ago.
>> If there happen to have been as many as four humans working on Navier-Stokes at any given time since the year 2000, then that's more human-years applied to the problem than agent-years.
My bad for not showing my work and inadvertently leading you down the garden path, but the "~100 agent-years" calculation goes like this:
36,666.7 agent-days / 365 days = 100.5 agent-years.
That's what you get for working 24 hours a day, 7 days a week, 365 days a year. Realistically speaking, that's not a work schedule any human can follow.
It's hard to make a realistic estimate because normally even a very dedicated mathematician will not be working exclusively on one problem all their waking time, or even all their working time. But, let's ignore this and assume a pretty standard work schedule of 8 working hours, five working days a week, and 52 working weeks a year.
Now, that's:
8 hours * 5 days = 40 working hours a week
40 hours * 52 weeks a year = 2080 hours a year
880,000 agent-hours / 4 humans = 220,000 hours per human
220,000 hours per human / 2080 hours a year = ~105.8 years
To clarify, that's how I estimate the number of years it would take a mathematician to do a quarter of the work of the 10k OpenAI agents if that mathematician worked only on solving Navier-Stokes and did nothing else in their entire career.
That's just not a realistic work schedule for any human. You can adjust the working hours if you want but I don't believe you'll get any realistic estimate. Don't forget that most academics' careers last around 30 years from PhD to Professor Emeritus. If you want a more realistic estimate of how much time it would take how many humans to do the work of the 10k OpenAI agents, you can start from that assumption and work your way up from that.
>> That would be a more convincing argument if the AIs, like the humans, had been around and trying to solve those problems for the last 2k years. However, as you might have noticed, the state of the art in AI was rather primitive 2000 years ago.
Sure. But the thing is agents can run 24/7, 365/365 in parallel and as you see above they can cover 2000 years of human work in much less time. I'm not going to estimate how much because the only bottleneck is the amount of compute and money that an AI company wishes to spend, and that depends on their motivation to solve a particular problem. However, with sufficient motivation 2k years of human research (keeping mind that's not 2k years of continuous work) can be covered in a few ... months? Probably.
> My bad for not showing my work and inadvertently leading you down the garden path
The problem isn't that you didn't show your work, it's that your work was wrong.
I entirely agree with your calculation that 10k agents for 88 hours is about 100 agent-years if we assume 24/7/365 operation. That's not what I was disagreeing with.
But then you said "How many human-years ...?" followed by estimating not the number of human-years that have gone into the problem but merely the number of years.
You can compare elapsed years for humans (26) and elapsed years for AI systems (about 0.01). You can compare agent-years (about 100) and human-years (26 times the average number of humans working on Navier-Stokes at any given time). Either of those is defensible.
But it makes absolutely no sense at all to compare agent-years for the AIs and elapsed years for the humans. Which is what you did.
If a typical human mathematician works 2000 hours a year (actual human mathematicians generally find that they can't do 8 hours a day of focused hard intellectual work, but I think we should count some of their "percolation time" too) then that's about 6 human-years per mathematician. So to get the same amount of mathematician-work as agent-work the average number of mathematicians you need to have been on the job is about 100/6, or about 16.
So when you wrote
> How many mathematicians have worked on the problem? 10k? Not likely.
the 10k figure was a total irrelevance. The number it would actually have to have been is about 16.
(My earlier "as many as four" ignored the fact that humans don't work 24/7/365, as you point out. But my point is that however you slice it the relevant number is more like four than it is like 10,000.)
My guess, for what it's worth is that that is roughly the order of magnitude of the number of human mathematicians working primarily on things that could be classified as "trying to make progress toward resolving the Navier-Stokes problem" during that time. I wouldn't be surprised if the actual figure were 3x bigger or 3x smaller. It probably depends on how broadly you interpret "trying to make progress toward resolving the Navier-Stokes problem", and one important difference is that all those human mathematicians leave behind them a trail of papers proving things that, whether or not they end up on the path to Navier-Stokes, may turn out to be useful later, whereas if OpenAI's agent swarm proved a lot of useful theorems along the way most of them never got published.
I don't, of course, disagree that it's possible for an AI company to put a lot of AI agents to work on a problem, but I'm not sure how that makes what they can do less impressive. The fact that you can do that has always been a major part of why AI could be such a big deal. "A country of geniuses in a datacentre" is the kind of thing people have said; we aren't quite there yet, but the "country" part is as important as the "geniuses" part.
I'm sorry but I'm not sure I understand your argument. I think you're saying I'm comparing apples to oranges. I'm not: I'm comparing apples to apples and oranges to oranges. These are two different questions:
>> But how much time have human brains spent working on the problem in either of those time periods? How many mathematicians have worked on the problem? 10k?
So neither 10k humans worked on Navier-Stokes, nor has any human spent a century of non-stop work on it.
But I could have made the point more clear maybe.
>> I don't, of course, disagree that it's possible for an AI company to put a lot of AI agents to work on a problem, but I'm not sure how that makes what they can do less impressive. The fact that you can do that has always been a major part of why AI could be such a big deal. "A country of geniuses in a datacentre" is the kind of thing people have said; we aren't quite there yet, but the "country" part is as important as the "geniuses" part.
Yes, I see your point, but those are not geniuses. Grigori Perelman proved the Poincaré conjecture alone, though as he has emphasised his work was based on advances made by others, particularly Richard S. Hamilton. That we can call a genius: a single man who solves one of the most interesting problems in all of mathematics building on the work of his predecessors. 10k agents that search blindly and find a result by luck (or by stealing it), I don't agree we can call "genius". That's what I call "brute force". Anyone who wants to call OpenAI's agents "a country of geniuses" has first to deal with the fact that they look a lot like monkeys on typewriters.
The question you asked was whether the AIs could prove the thing "in a vacuum", without making any use of previous human work. "Close to being solved" is a statement about all the previous human work that mathematicians might have used to do it in five years.
I am a mathematician, though I haven't been in academia for many years, and I would indeed be surprised if there were a lot of mathematicians who "just memorize proofs". Do you really mean that, and if so what's your evidence for it? (And: how do you suppose it is possible for someone who merely memorizes proofs without caring what they are about to prove new things in their field?)
The article says "A single firm, Irregular, is responsible for hacking done by all three companies" but I can't see anything in the article that actually justifies this claim. The nearest to that is the sentence immediately after that one: "Anthropic disclosed that Irregular was responsible for creating the tests ...". This is not, in fact, the same thing.
(Especially as, as aesthesia mentions, the article just happens not to mention that by "hacking done by all three companies" it doesn't mean, e.g., the most famous recent examples of such hacking: Irregular wasn't involved in the OpenAI/HuggingFace incident.)
So, so far as I can tell, the story is: OpenAI and Anthropic make AI models. Irregular does AI model evaluations. In some of Irregular's model evaluations, in which supposedly-sandboxed models attempted to break into simulated targets, the models got out of the sandbox and did bad things in the external world.
The article talks about "firms which instruct AI models to commit cyberattacks", which is a very neat bit of dishonest framing. It's true, in a sense, that Irregular instructed the models to commit cyberattacks -- inside their sandbox, against fictitious hosts. It's also true that the models actually did commit cyberattacks (e.g., the Hugging Face incident, though once again the attacks described by the article don't actually include this one). But it's not at all true that Irregular instructed the models to do anything like the bad things they actually did.
The article says '[Anthropic's] later disclosure shows that exactly zero percent of the agents went "rogue"'. Once again, the disclosure does not in fact show that. It shows that one variety of going-rogue could have been prevented by telling the models explicitly "this thing is real, not part of any kind of test, leave it alone". That is not the same thing.
The article claims that 'In the wake of these attacks, Anthropic and Irregular have deployed a swarm of AI Safety influencers paid by Anthropic-connected foundations to distract from their culpability and towards the baseless “rogue agent” theory.' It offers no actual evidence for this.
And the article seems very keen to highlight links between the companies involved and "effective altruism", though it is -- I assume deliberately -- rather vague about whether it's saying "of course we all know that EA is evil, so that shows that these companies connected to EA are evil" or "this incident shows how evil EA is".
The "Effort News" website has a number of other look-at-the-scary-Effective-Altruists stories on it. They also strike me as rather bullshitty.
... And then I look a bit further, and I see that Effort News's "about" page says "It all started when I was experimenting with using AI for financial auditing. I found stories that were crucial to the public’s right to know, including several of the stories now available at /investigations. I knew we had to sprint to the launch and launch a publication, directly applying this technology." and "The scope of what we can investigate has massively expanded, because we can chase 1,000 misses for one hit. But the final product cannot be slop. There’s plenty of slop on the internet. The way to surpass that, and what really matters, is manual curation and review of every finalized story."
Manual curation and review? I think the people behind Effort News are admitting that this is AI-generated "journalism". I expect that one day AI systems will be trustworthy journalists, but I personally am not very convinced that that day has yet come. And I don't see much reason why I should trust Brian Chau, the guy behind Effort News, to be doing everything possible to make his AI systems trustworthy journalists. It looks to me as if maybe they've been given instructions along the lines of "dig up things that make Effective Altruism look bad" for some reason.
(I don't mean to imply that EA is their only target. It's just one that jumped out at me.)
> Irregular wasn't involved in the OpenAI/HuggingFace incident.
It was [1]. It's understandable that you assumed it wasn't because the article didn't cite the sources on this claim. I agree with the rest of your points.
I suspect EdwardDiego is referring to the brouhaha about whether OpenAI's training for the model that produced the alleged solution to the Millennium Problem about the Navier-Stokes equations was trained on material that included conversations Tristan Buckmaster and Levent Alpöge had had with earlier OpenAI systems.
I think there's a bit less to that than meets the eye. Yes, OpenAI's result builds on human work. It's possible that it builds on more human work than OpenAI admitted. But even if we suppose that everything Buckmaster and Alpöge did (which, btw, was itself very heavily LLM-assisted/generated work) was a necessary precursor to what OpenAI released, it's still the case that OpenAI's clankers completed the solution and Buckmaster and Alpöge didn't.
My understanding from what Buckmaster has written about this is that the deep mathematical ideas behind their work (and presumably OpenAI's) are due to Córdoba and Martínez-Zoroa. Those ideas are in the published literature, and human mathematicians and AI systems alike are allowed to use them, and doing so doesn't mean they didn't actually do something impressive. Mathematicians build on one another's work; that's how mathematics progresses and always has been.
It may very well be that OpenAI's announcement has a serious problem of professional ethics, especially as their first version of it didn't even list Córdoba and Martínez-Zoroa in its references. (On the specific question of what if anything they learned from B&A's work before that was published: OpenAI are now claiming that after investigating carefully they are confident that the model was not trained on anything Buckmaster and Alpöge did after early July. B&A had been working on this thing for much longer than that. However, on Buckmaster's account of things it wasn't until mid-August that they got beyond what he calls "preliminary results".)
But! The results of B&A were themselves largely AI-generated. (From Buckmaster's statement: "on August 15th, we obtained the blow up results, with smooth forcing, for both Boussinesq and Euler. I can say the first LLM generated proof Levent sent me was the most horrendous I have ever read; we verified it on Lean on August 22nd. Since this point, we have been working around the clock to understand this proof and turn it into something readable." That is: the LLMs found the proof, and B&A had to work to understand what the LLMs had done. It's not that humans did the thinking and AIs just did the gruntwork. (Except in so far as one might want to give all the credit for Real Deep Cleverness to C&MZ.)
And! What OpenAI say their model has proved goes well beyond what B&A did.
I don't see any way of slicing this that makes it unreasonable to say (unless it turns out that there's an error in the proof -- unlikely, given that it comes with Lean verification, but there have been misformalizations and Lean bugs in the past and there surely will be in the future) that AIs solved the N-S problem. No, they couldn't have done it without the work of C&MZ, but again: important mathematical work almost always builds on earlier important mathematical work, that's just how it is. Yes, if OpenAI are lying through their teeth their model might have had early access to B&A's ideas -- but it seems like most of the B&A work was actually done by AI systems anyway.
It is (I think -- I am not an expert and in particular I have not so much as looked at OpenAI's publication) reasonable to say that the deepest ideas here came from humans, and that it was already widely expected that the N-S problem would be solved in the not-impossibly-distant future in something like the way it has been. So, sure, what the AIs have done here is much less impressive than if they'd settled the Riemann Hypothesis or (probably even harder) PvNP. But it's still a resolution of a famous mathematical problem that any human mathematician would have been very proud to have achieved.
reply