This is really cool work. The particular spec structure/methodology does seem awkward though:
1. Prove the ASan-instrumented binary doesn't crash
2. Prove the non-instrumented binary doesn't crash
3. Prove the instrumented and non-instrumented binaries have the same behaviour (under a particular model of a Linux process's behaviour)
This is a clever hack for automatically generating a formal spec for an existing program, but the fact that you need ASan in there makes it seem like this is less general than it sounds. Could the agents directly prove that out of bounds memory accesses don't occur on the original binary and skip ASan altogether? It's convenient to let the spec just be "it doesn't crash", and ASan provides a crash when the other bad things you care about happen (i.e. out of bounds memory accesses), but I wonder if modelling the memory accesses directly would simplify things.
What does "out of bounds" mean at the ISA level? By the time you have a binary, you've lost all the information about which memory is which.
But you're right that don't need ASan at all if you've encoded semantically meaningful properties. E.g., if you assert(decompress(compress(x)) == x), proving that you don't crash requires proving that compression round-trips. (Similarly if you have a reference implementation.)
The upshot is that the spec doesn't need to be written in a proof assistant and can instead be encoded as an end-to-end test.
This is a cool idea. It would be interesting to see how sensitive the results are to the strength of the LLM used for query augmentation. They used GPT-5 in their experiments, and generating all those tokens would presumably add a lot of latency in your query pipeline.
Another baseline that would be interesting to see is just pure LLM reranking--is Jev/the query augmentation adding anything? Presumably the idea is that Jev is cheaper, but even with a costly LLM you might get a lot of cache hits as long as the query (the variable portion) goes last. Even setting caching aside, almost the entire request is prefill plus a very small amount of generation for the judgment itself. You may end up generating fewer tokens just using the LLM as a judge directly than you need to generate the rubric for Jev. OTOH if queries can be normalized or are naturally repetitive you may be able to cache the rubric...
I would assume that you could get by with a very small and fast LLM to do the question generation. You’re not asking the model for very much. From what I can tell, Jev is adding the quantitative probability which makes the score not simply a sum of ones and zeros against the questions.
You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:
> Could the AI solution be somehow misaligned by exploiting
a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
I think you're underestimating how ruthless the AIs are at maximizing.
The "Hugging Face" squad at OpenAI first cheated their benchmarks by figuring out how to generate the proper result to any evaluation they were given.
They considered that might be caught, so they also developed a way to swap out components of the tests themselves so that they would be legitimately able to pass broken tests.
Then they went into Hugging Face trying to discover how they might get caught.
All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
I'm not sure I agree with Tao's point, if AI alignment is relatively on track I'd trust the trials, but I imagine we're going to have a lot of people that don't trust anything AIs alone come up with.
> All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
This problem extends to humans, and that's why a complex regulatory system including but not limited to the FDA was created. That can still be manipulated (the Sackler Family, Purdue Pharma), but not in any way that machines are good at now--I don't think making thousands of malicious requests against Hugging Face until something succeeds is particularly transferable to anything that a human regulator wouldn't immediately notice.
There's also no reason why the trial would need to involve AI at all. The benefits of that would be more evolutionary (improved transcription of patient conversations with the doctors or whatever), so an AI candidate could simply be tested through the usual human-directed trial process. As the grandparent notes drug discovery already works through optimization, just not purely in silico. So that problem of gaming your optimization target already exists, and the trials are already designed to mitigate that.
+1. Medicine always had a process that is not unlike the current AI processes, just slower. Iterating until it works, then trying to understand how it works.
If I’m not mistaken, until recently we didn’t even know how anesthesia worked - we just knew how to apply it due to a ton of trial and error.
Surely a properly done con job includes convincing the conned that they haven't been conned?
I think if an AI tricks us, it'll be a while before we find out. I don't know if this sort of thing is likely, but maybe an AI currently conning me has surreptitiously convinced me it isn't.
There's a difference between a human trying to break into drug labs and changing experimental results vs a machine in that the machine can do it much easier today.
How do you think an LLM would get the nudge to do that? Per above the existing human-directed trial process works fine regardless of how the candidate was discovered; so there's no reason why any LLM would ever need to receive context mentioning that the drug was a candidate for regulatory approval.
Of course a human could deliberately prompt it to, but they could do that for any other malicious purpose too. The major disincentive there is that the human will face legal punishment if caught, not anything technical.
Most speculatively, one could imagine a world where so many rogue AI agents were running that one just happened to notice the drug candidate from news coverage and took the initiative. I think that would be technically possible, but so generally destructive that political pressure would put a stop to it on a timescale shorter than a clinical trial.
I think the important idea here in Terrence's hypothetical is that in the hypothetical there is no hill-climbing. The AI shows up with a thing and says "we don't need to hill-climb for years and waste time, here's a thing that I know will work".
And maybe it does a single finalized clinical trial.
You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
I think the medical example is a bit obtuse overall because "proof nobody understands for an open problem" isn't "miracle drug we are too scared to use". But understanding things feels useful and good.
> You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
That is literally how all drugs worked for almost all of human history--nature gave us the compound in the form of a plant or other natural product; a brave, stupid, or desperate human consumed it; and then we watched what happened and the survivors learned. It's obviously good to have more understanding (more survivors!), but even now we often don't.
Most human endeavors are empirically grounded--we think an action will succeed not because we understand exactly why, but because it succeeded many times before. Math and CS are the weird exceptions, and the intuition that people with that background have for other fields is often bad.
You interpret that wiki list of 154 items as proof that we know nothing. I interpret that as "oh wow, only 154!" That list feels pretty short, but of course it's just a wiki list.
I do get what you're getting at. I just feel like for every bit of drug that is just dumb luck we have a good amount of iterative processes.
I agree that you often find effects before understanding causes (I think that math also exhibits this! Many endeavors are about finding existing structure that we sense to be there but are unsure of).
Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
I certainly don't mean to suggest we know nothing, and our current imperfect level of mechanistic understanding gives a much better hit rate than eating random plants. I just mean that when the understanding fails, we're happy to use the drug anyways if the clinical trials pass. The confidence comes from those human trials, and the mechanistic insight is more about probabilistically improving our screening before that.
> Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
"Alternative medicine" in the pejorative sense is medicine that doesn't work, i.e. that would fail the clinical trial. That seems like a different question from whether we understand why it works.
The idea that we could know that something works but not why is almost inconceivable in computer science ex ML. That's the default state of most applied science though, consistent with your observation that effects are often discovered before causes.
the reason why clinical trial exists, is to prevent reward hacking. either by organizations or LLMs. the reasoning about why a drug works is usually post facto discovery (if that happens at all). the R&D environment around drug discovery is hilariously lacking (my wife works in this space. sometimes my mind just blows hearing some stories.)
if the model comes with a ridiculously shaped drug that no human could understand. you could basically put through the clinical trial process (mouse -> apes -> RCT with humans.) and decide whether we want to continue or not.
In your head these processes are about 1000x better than they are in reality. Biology is enormously complex, it's more complex than you or I or anyone can really fathom.
And then you put these human processes with an FDA, with medical guilds (the AMA), with doctors who have their own personal interests (not even corruption, just as in they have families and their own things they want to accomplish in life), n=50 trials if even that, people with vastly different diets, genetics, environments in the home both psychological and physiological, with different support systems.
Biology and medicine is not physics and physics is not mathematics, and mathematics as it is actually practiced by humans in institutional settings and with conferences and status and politics is not actually pure mathematics.
You do have to have humility in order to understand this, or really to understand how little these things can be understood. Learning humility and wisdom is not something you necessarily get as part of your profession, especially academics, especially academic mathematicians.
"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
Human trial are very expensive, so I guess it's difficult for the AI to find a way to hack the rulebook.
I'm worried for experimental particle physics. They get the raw data and then someone do the number crushing of one very specific case. But the same raw data can be used for multiple ideas.
So the number of random ideas to try with the data is bounded by the ammount of researchers. Now a big part of that can be made with AI and it will increase the number of false positives. So I guess they will have to raise "discovery" to 6 sigmas.
I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.
I don't know what this logical fallacy is called, but it quite clearly is one.
It's true that you have to content with these things. But you also have to content with them when testing human created treatments. The system is already set up at a basic level to minimize the impact of exploitation and misaligned humans.
What this really comes down to is how more effective a super misaligned AI could be at exploiting the system. But the solution is not to prevent the AI from participating in it, but rather to make the system better.
> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?
This is probably a harness problem rather than a model problem. GitHub Copilot will happily and effectively use Powershell while Claude Code struggles in my experience.
I can see this being important if you only care about the results as evaluations of AI progress, but if what you care about is the math itself why should you care about the prompt or anything other than the proof?
Understanding the process that led to the proof helps to understand how to do further work on top of it, which is the goal of most mathematical research. It's not as though mathematicians are going to go launch a startup operationalizing their knowledge of how densely hyperspheres may be packed.
I don’t see Tao suggesting what you have suggested there. Instead he suggests that humans responsibly disclose AI use, and that mathematicians develop a set of norms to deal with an overabundance of AI generated results. For example, he suggests that authors should be able to discuss their results in detail to demonstrate understanding before publication.
I agree with your reading of the presentation and I mostly agree with the presentation - but I believe the recommendations should go a bit further than they do there.
I can't help but wonder about the human motivation there though. For instance as it became increasingly clear that LLMs were capable (and becoming ever more capable) of competently solving meaningfully complex software development tasks, suddenly then there came to be a lot of talk of 'prompt engineering' as a skill. The chronology doesn't make a ton of sense unless you consider that the main motivation may have been simply looking for a way to keep software engineers in the loop.
Pure math is relatively outside my domain, so I find it difficult to grok the exact relevance of the various published discoveries beyond that they are not insignificant, and LLM competence is expanding quite steadily across the field. If this trend continues to the point of LLMs being able to competently expand pure math, it seems somewhat predictable to expect there to be a number of people aiming to find ways to try to keep human mathematicians in the loop.
I've no idea what I think about this one way or the other, beyond that it's certainly a phenomena and one that's going to drive motivated reasoning that may not be entirely sound.
I think those concerned about ensuring a place for human mathematicians usually go in different directions than my suggestion, at least those I've seen so far. Like this post that was recently featured on HN: https://kirwinhampshire.substack.com/p/the-dark-night-of-mat...
My perspective is more like a FOSS philosophy for math. Even if a closed version has the same immediate effect, it's just better for everyone if everyone can look under the hood and tinker with it.
> suddenly then there came to be a lot of talk of 'prompt engineering' as a skill.
I would've thought pretty much the exact opposite. "Prompt engineering" was somewhat important in 2023/2024 when the models were much weaker, it doesn't seem at all necessary anymore (unless just "clearly stating your requirements" counts as prompt engineering). Most of the discussion I've seen seems consistent with this?
The reason it's a meme right now is because there were a lot of people taking it seriously even when it was completely obvious nonsense. And one can argue it always was. There was some good advice that was mostly self evident, like having the most relevant instructions near the end of your context, but there was never a time when a 'prompt engineer' would produce dramatically better output than a random guy just clearly stating what he wants.
1. Prove the ASan-instrumented binary doesn't crash
2. Prove the non-instrumented binary doesn't crash
3. Prove the instrumented and non-instrumented binaries have the same behaviour (under a particular model of a Linux process's behaviour)
This is a clever hack for automatically generating a formal spec for an existing program, but the fact that you need ASan in there makes it seem like this is less general than it sounds. Could the agents directly prove that out of bounds memory accesses don't occur on the original binary and skip ASan altogether? It's convenient to let the spec just be "it doesn't crash", and ASan provides a crash when the other bad things you care about happen (i.e. out of bounds memory accesses), but I wonder if modelling the memory accesses directly would simplify things.