That was an antibody against human CD28, and I'd heard speculation at the time that the monkey trials failed to predict the human response due to differences between monkey and human CD28. That was apparently not correct, since "the sponsor had not only determined the relevant extracellular sequence (which is 100% identical to the human one) but also the binding affinities of the mAb, which were identical for human and monkey". That author goes on to propose a different explanation, though I don't know if any explanation is generally accepted.
It's funny to recall this in the context of Terence Tao's question of whether you'd take an AI-designed drug with no mechanistic insight. The sponsors of this trial had reason to believe they had strong mechanistic insight, but they were wrong. A pharmacologist from the nineteenth century would have probably run the trial more safely (lower starting dose, one participant at a time) without that false confidence.
> “The Court finds that because the ALPR systems Alaniz used to search Kyle’s historical location information intruded on her reasonable expectation of privacy in the whole of her physical movements, it was a search under the Fourth Amendment. Based on the information in the record, the only reason Alaniz conducted that search was because he saw her license plate was from California. That search was not supported by probable cause, and it was done without a warrant in violation of Kyle’s Fourth Amendment rights.”
If the cop had obtained a warrant before searching ALPR data, then the search would have been legal under this judge's standard. But the cop (a) didn't obtain that warrant, (b) probably couldn't have obtained it quickly enough to decide whether to follow her until she committed a traffic violation, and (c) probably couldn't have at all since he didn't have probable cause. The question of what level of ALPR search requires a warrant is evolving rapidly now, in response to both case law and new legislation (e.g. Connecticut SB397).
Kyle also disputes whether the ALPR search found anything that justified the detention. If the ALPR search was itself illegal then that's irrelevant though.
You are correct that the judge ruled the stop was "justified at its inception"; but the section immediately following your quote is titled
> Alaniz did not Have Reasonable Suspicion to Extend the Traffic Stop.
That's the newsworthy part. For example, Kyle stated that she had the marijuana only after Alaniz detained her so that he could attempt to search the car without her consent. If that detention was illegal then her statement isn't admissible, and that's what the judge ruled here:
> Because the government does not identify any adequate exception to the exclusionary rule, all evidence obtained after Alaniz initiated the ALPR search must be suppressed as fruit of the poisonous tree.
ETA: And while the initial stop was lawful, the grandparent comment is probably correct that it wouldn't have happened without Flock. Alaniz performed a Flock search that included "traffic violation" as the reason, but he hadn't yet observed a traffic violation at that time. Based on that ALPR history he correctly suspected drug trafficking, and followed the car until he observed a traffic violation that justified the stop (but not the subsequent fishing expedition).
> All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
This problem extends to humans, and that's why a complex regulatory system including but not limited to the FDA was created. That can still be manipulated (the Sackler Family, Purdue Pharma), but not in any way that machines are good at now--I don't think making thousands of malicious requests against Hugging Face until something succeeds is particularly transferable to anything that a human regulator wouldn't immediately notice.
There's also no reason why the trial would need to involve AI at all. The benefits of that would be more evolutionary (improved transcription of patient conversations with the doctors or whatever), so an AI candidate could simply be tested through the usual human-directed trial process. As the grandparent notes drug discovery already works through optimization, just not purely in silico. So that problem of gaming your optimization target already exists, and the trials are already designed to mitigate that.
+1. Medicine always had a process that is not unlike the current AI processes, just slower. Iterating until it works, then trying to understand how it works.
If I’m not mistaken, until recently we didn’t even know how anesthesia worked - we just knew how to apply it due to a ton of trial and error.
Surely a properly done con job includes convincing the conned that they haven't been conned?
I think if an AI tricks us, it'll be a while before we find out. I don't know if this sort of thing is likely, but maybe an AI currently conning me has surreptitiously convinced me it isn't.
There's a difference between a human trying to break into drug labs and changing experimental results vs a machine in that the machine can do it much easier today.
How do you think an LLM would get the nudge to do that? Per above the existing human-directed trial process works fine regardless of how the candidate was discovered; so there's no reason why any LLM would ever need to receive context mentioning that the drug was a candidate for regulatory approval.
Of course a human could deliberately prompt it to, but they could do that for any other malicious purpose too. The major disincentive there is that the human will face legal punishment if caught, not anything technical.
Most speculatively, one could imagine a world where so many rogue AI agents were running that one just happened to notice the drug candidate from news coverage and took the initiative. I think that would be technically possible, but so generally destructive that political pressure would put a stop to it on a timescale shorter than a clinical trial.
> You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
That is literally how all drugs worked for almost all of human history--nature gave us the compound in the form of a plant or other natural product; a brave, stupid, or desperate human consumed it; and then we watched what happened and the survivors learned. It's obviously good to have more understanding (more survivors!), but even now we often don't.
Most human endeavors are empirically grounded--we think an action will succeed not because we understand exactly why, but because it succeeded many times before. Math and CS are the weird exceptions, and the intuition that people with that background have for other fields is often bad.
You interpret that wiki list of 154 items as proof that we know nothing. I interpret that as "oh wow, only 154!" That list feels pretty short, but of course it's just a wiki list.
I do get what you're getting at. I just feel like for every bit of drug that is just dumb luck we have a good amount of iterative processes.
I agree that you often find effects before understanding causes (I think that math also exhibits this! Many endeavors are about finding existing structure that we sense to be there but are unsure of).
Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
I certainly don't mean to suggest we know nothing, and our current imperfect level of mechanistic understanding gives a much better hit rate than eating random plants. I just mean that when the understanding fails, we're happy to use the drug anyways if the clinical trials pass. The confidence comes from those human trials, and the mechanistic insight is more about probabilistically improving our screening before that.
> Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
"Alternative medicine" in the pejorative sense is medicine that doesn't work, i.e. that would fail the clinical trial. That seems like a different question from whether we understand why it works.
The idea that we could know that something works but not why is almost inconceivable in computer science ex ML. That's the default state of most applied science though, consistent with your observation that effects are often discovered before causes.
That's an unrealistically idealized view of drug development. The human use of drugs long predates anything resembling a modern concept of a biochemical mechanism. Nobody knew how morphine relieved pain, but they knew that it worked and they used it.
Today we'll generally have a proposed mechanism, but it's not necessarily correct. Acetaminophen is among the oldest and most commonly used synthetic drugs, and its mechanism is still debated. This isn't usually cause for any special concern, since our confidence in the drug's efficacy and safety comes more from animal or human trials than from mechanistic understanding. Serendipitous discoveries during human trials are still common; the first inkling that Viagra might treat ED came not from any "first seminal paper" but from the volunteers who were testing it for angina.
Clinical trials are regulatory matters. Your suggestion that it could be "unclear who conducted the phase 3" is very strange--the FDA knows who filed the application. Of course that filer could have committed fraud, but I see nothing in the slides to suggest that was the concern here. If it was, then the solution would be a non-fraudulent phase 3, not anything related to mechanistic insight.
>>Your suggestion that it could be "unclear who conducted the phase 3" is very strange--the FDA knows who filed the application
This means the drug has gone through the actual drug development lifecycle- Terrence is not talking about this.
>Acetaminophen is among the oldest and most commonly used synthetic drugs, and its mechanism is still debated
You answered your confusion; it's an extremely old drug (1950s) and back then we didn't really know anything about pathway targeting. The drugs that we have developed in the recent decades (post 2000s); we generally start from the target; the same story that i wrote in the main comment - so we do know about the mechanisms and the pathways that are targetted, down to the genetic level sometimes.
So if a new drug showed up on the market today which actually has gone through the above lifecycle; we could with a minimal effort put together 5-6 scientists and get a consensus on how the drug is actually working.
No pharmaceutical is testing drugs today which have no scientific theory or reasoning or seminal papers behind it.
> This means the drug has gone through the actual drug development lifecycle- Terrence is not talking about this.
I don't understand what you mean by this. Nobody is conducting phase 3 clinical trials of a new drug except as part of that regulatory drug development lifecycle, to gain legal permission to market the drug. The idea that it could be "unclear who conducted" a >$10M trial closely supervised by multiple government agencies does not make sense.
> No pharmaceutical is testing drugs today which have no scientific theory or reasoning or seminal papers behind it.
Even today, new drugs are routinely identified by screening with assays that give no mechanistic insight, as e.g. for ganaplacide. They found that drug by screening directly for whether the compound killed Plasmodium (the genus of parasites that cause malaria), by applying it to infected cultured liver cells[1]. That was probably a lot more expensive than something like a protein binding assay against a prior known target, but probably more fundamentally accurate. That also creates potential to discover an entirely new mechanism of action, without potential for cross-resistance with drugs developed against prior known targets, and that apparently happened here.
The first studies in humans were conducted around 2014[2], at which time I believe no mechanism had yet been proposed. Some progress towards a mechanism was proposed in 2020[3] and perhaps it's correct; but if it's later shown to be incorrect then nobody will care, and that in no way drove the discovery or initial human testing of the compound.
The bad cancer example is interesting to the extent it reveals that one of the world's top mathematicians is apparently unaware that applied science runs almost entirely on half-understood semi-empirical methods. That's most true for medicine, given the extreme complexity of human life; but if you look at a modern SPICE model for a transistor, a device that we claim to understand at the level of subatomic particles, then you'll find it's full of curve fits. As an engineer, I find that perfectly normal. My job is to use all the tools at my disposal to meet some human desire (to not die of exposure, for a slightly thinner phone, etc.), and whatever fundamental understanding I might have is only one tool to that end.
My impression is that since pure mathematics doesn't attempt to meet those human desires they need some other objective, and that objective is human understanding. The loss of that is thus felt more heavily than in other fields. We could say it's their problem, and they need to get over it just like the chess players did; but the outside implications are broader here, since mathematicians working in fields they themselves considered useless have so frequently been wrong--in Hardy's Mathematician's Apology, he gave number theory as an example of such a field, unaware of what the cryptographers would achieve just decades later.
It's possible that AI-generated pure math will continue this trend of delivering extraordinary unexpected societal value. It's also possible that the humans won't ever sufficiently understand that math, and the machines won't ever sufficiently understand human desires, and that connection won't be made. I've never met a pure mathematician who considered those downstream applications to be an important contributor to their motivations; but as AI-generated math contributes to the argument to allocate a large and increasing share of GDP to datacenter buildouts, that question of whether downstream value requires human understanding seems pressing.
Yes, human understanding is central to his argument. My interpretation of his slides:
The math problems AI helped solve were problems identified by humans in the past (lighthouses, for the field of math). They're problems meant to guide the field: working towards solving the problems brings new ideas to light, and possibly steers us into new areas we wouldn't have considered in the past. More is considered, and possibly more is discovered along the way. (It's very difficult to validate that statement, and so there's plenty of room for argument, and is the heart of the discussion)
Using AI to take a shortcut to the lighthouse creates a survivorship bias. All attention is spent on proving solved problems: less is considered, and certainly less is understood.
Academia was set up for the old journey to the lighthouse, and if we don't consider carefully the new journey to the lighthouse, we'll lose all ability to create new lighthouses (or steer ships).
He seems to suggest it's more valuable for humanity to integrate AI into the academic pipeline, versus spending all attention on solving existing open problems.
> but if you look at a modern SPICE model for a transistor, a device that we claim to understand at the level of subatomic particles, then you'll find it's full of curve fits.
In both grad school and professionally I've worked in this area. You are not correct.
At the research level, they have simulations based on real physics (not curve fits) that will determine the transistor properties quite accurately. It's computationally expensive - a SPICE model needs the answer in milliseconds, and the simulation takes a few hours to get the answer.
A SPICE model is essentially (mostly) simple classical equations for a MOSFET, coupled with curve fits to fix Nth order effects. A lot of the curves are still based on physics knowledge, but parametrized to make it easy to fit the data.
> At the research level, they have simulations based on real physics (not curve fits) that will determine the transistor properties quite accurately.
That's true, but that's why I restricted my statement to a "modern SPICE model". I said "modern" because the Level 1-2 FET models contain only parameters with known physical meaning. Subsequent levels improve accuracy, but some of the parameters they introduce do not have known physical meaning.
> A SPICE model is essentially (mostly) simple classical equations for a MOSFET, coupled with curve fits to fix Nth order effects. A lot of the curves are still based on physics knowledge, but parametrized to make it easy to fit the data.
I agree with that description, and that's the blend of theoretical understanding and empirical curve fitting that I was trying to capture with the phrase "semi-empirical". My point is that even for something as well-understood as a FET, we found that semi-empirical model to be more convenient for most practical purposes. As engineers we think that's fine (all models are wrong but some are useful, etc.), but pure mathematicians seem to find it unthinkable.
The difference between SPICE models and, say, drug discovery is for FETs, we have a lot of data to test the (fitted) model. We used to ensure it fit the real MOSFETs at different temperatures, dimensions, different drain/gate inputs, etc.
On top of that, since there is so much variance in transistors within one fab process, circuit designers ensure their designs work with a fairly wide margin. So the accuracy of the SPICE model is not really that important.
There is no equivalent in cancer treatments.
> As engineers we think that's fine (all models are wrong but some are useful, etc.), but pure mathematicians seem to find it unthinkable.
I'm sure they know how real engineering works (and they either admire or sneer). But of course, they cannot bear such a workflow for their own research.
Tao apparently didn't though, if he thought the absence of a known mechanism would discourage anyone from using an AI-discovered drug. I feel like a pure mathematician's worldview might be so focused on human understanding that he simply couldn't imagine that billions of people would already be taking drugs nobody understands. To fall from that to puzzling over LLM-generated proofs must be brutal.
SARS-CoV-2 has relatively low IFR, especially among younger people, and a mild case is easily dismissed as a flu or cold. Transmission is also heterogeneous, with most patients infecting no one while a few infect huge clusters; R = 2 absolutely does not mean one patient infects two, who infect four, who infect eight, etc.
> An important feature in this distribution, consistent with earlier findings (4), is that the majority (76%) of infected cases shows no evidence of onward transmission amongst any of their contacts.
So there's no reason to expect that the first cluster that generates enough sickness and death to attract official attention will occur near the point of introduction. If we look at North America, then it in fact did not--the introduction must have been at an airport or seaport, but the first large clusters were in nursing homes, church choir practices, etc. This is different from SARS-1 or MERS, which were deadly enough to make that cryptic spread much less likely.
Can you clarify your point? Crowded, poorly-ventilated wet markets like the Huanan Seafood Wholesale Market are quite different from American (or Chinese) supermarkets, and uncommon in the USA. American-style nursing homes are likewise uncommon in China, where it's more common for elderly parents to live with their children.
The conditions for a super-spreader event can arise in many different ways--before the contact tracers gave up, they found clusters in that wet market, those nursing homes, cruise ships, meat processors, migrant worker dormitories, and many other types of venues. I don't see why we'd ever expect the first cluster in the same type of venue in both countries, except by chance.
I'd further note that the H1N1 flu is widely believed to have gone extinct in the wild, then been reintroduced in 1977 from a laboratory source, probably through a failed vaccine trial:
> The reemergence was probably an accidental release from a laboratory source in the setting of waning population immunity to H1 and N1 antigens.19,31
This explanation is now printed uncontroversially in major journals, though at the time the WHO rejected it:
> Laboratory contamination can be excluded because the laboratories concerned either had never kept H1N1 virus or had not worked with it for a long time.
Nobody expected natural zoonotic spillover near Wuhan, and Dr. Shi's lab was not situated with that in mind. In her own words:
> We have done bat virus surveillance in Hubei Province for many years, but have not found that bats in Wuhan or even the wider Hubei Province carry any coronaviruses that are closely related to SARS-CoV-2. I don't think the spillover from bats to humans occurred in Wuhan or in Hubei Province.
The virus doesn't seem to have been engineered from any previously known sarbecovirus, but the WIV had the largest collection of novel sarbecoviruses collected from nature in the world, and had proposed in DEFUSE to engineer them in collaboration with Dr. Baric at UNC. DARPA rejected that proposal, in part for safety reasons, but the WIV was technically capable of doing the work alone.
> Dr. Shi's lab was not situated with that in mind
Never said that, labs are close to places with high viral reservoirs because it is easier to get animal samples to study, obviously, not to guard against outbreaks. Neither one "expects" such an outbreak, obviously. The fact that it is a tail event does not mean it is arbitrary, though, ie that it can just happen anywhere with equal chance.
The other arguments have been answered 1 million times and they are not worth even discussing about anymore. They only exist as justification for somebody really wanting certain conspiracy theories to be true.
> Never said that, labs are close to places with high viral reservoirs because it is easier to get animal samples to study, obviously,
You are continuing to imply that Dr. Shi's lab was "close to places with high viral reservoirs", but nobody--lab leak advocate, zoonosis advocate, or otherwise--thinks that's true. She collected "animal samples to study" from Yunnan and surrounding areas, >1000 km from Wuhan. For example, RaTG13 came from Tongguan (thus TG), in Mojiang County. The location of her lab did not make those samples "easier to get".
There are probably lots of other labs that are deliberately placed near natural reservoirs of the organisms that they study, but Dr. Shi's was not. If anything, she thought those sarbecoviruses were less abundant near Wuhan than in some random location in China, not more.
She could be wrong, but no evidence has yet emerged that she was. Even among advocates for theories of natural zoonosis, the pathway is usually proposed to be smuggling of an unknown intermediate host rather than any natural reservoir in Wuhan, thus the attention on the wet market (since a similar pathway probably did explain SARS-1).
As to the rest, what evidence do you think could distinguish one natural novel virus from a laboratory chimera of two novel viruses? I think it would be more helpful to explain the basis for your belief than to smear mine as a "conspiracy theory".
https://www.bbc.com/news/magazine-35766627
That was an antibody against human CD28, and I'd heard speculation at the time that the monkey trials failed to predict the human response due to differences between monkey and human CD28. That was apparently not correct, since "the sponsor had not only determined the relevant extracellular sequence (which is 100% identical to the human one) but also the binding affinities of the mAb, which were identical for human and monkey". That author goes on to propose a different explanation, though I don't know if any explanation is generally accepted.
https://pmc.ncbi.nlm.nih.gov/articles/PMC2990150/
It's funny to recall this in the context of Terence Tao's question of whether you'd take an AI-designed drug with no mechanistic insight. The sponsors of this trial had reason to believe they had strong mechanistic insight, but they were wrong. A pharmacologist from the nineteenth century would have probably run the trial more safely (lower starting dose, one participant at a time) without that false confidence.
reply