Just to say the obvious, the idea that it's unethical to distill from AI models, but totally ethical to train on copyrighted source material is … definitely a take that the frontier labs have encouraged.
Fair point. But Meta also has a vested interest in keeping terms of service strong, so it wouldn’t really do for them to violate Anthropic’s and then go to court over it. Meta would then be in a lose-lose scenario, all over an open weight model that is a money sink to begin with.
Has anyone run a test of including some shibboleth or canary phrase or assertion in a chat, enabled for training, and seeing if it turns up later as something a model "knows"? I'd be curious to understand how that works even in a toy-level model, and if there is anyone consciously testing that process with the frontier lab offerings.
My naive instincts would be that it seems unlikely that a single chat transcript would leave much of an impression on a model, but I'd be very curious to learn how that works.
250 documents ingested from somewhere is enough to become part of the knowledge of a model of arbitrarily large size.
I would expect that a good idea that fits in a framework that is already being ingested would be more easily taken up than some random thing unassociated with anything else. Could that go down to a single transcript? If the model is consciously focusing on everything X related, quite possibly.
Thats not what they are asking. This paper is discussing documents in the training dataset poisoning the LLM for malicious behavior. This person are asking if anyone has deliberately put something in a private chat (presumably with retrain on my data turned off), to see if they can get it to leak across sessions from distinct users. I am positive this happens but I have not seen the proof. I also want to know the answer to this question.
* with train on my data turned ON, yes. Though OFF would of course be even more notable!
Thank you – the non-adversarial reproduction paper ( https://arxiv.org/abs/2411.10242 ) nails it – from chat, to training corpus, to subsequent model. Though in my hasty read, it is not entirely clear whether the snippets it finds are nonces, i.e. present exactly once in the internet.
I presented the research that I knew was somewhat relevant. Then made it clear that that wasn't what was being asked, and why my expectation is what it is.
Ah i see what you were saying now, you were saying session specific poisoning is more likely to work if it was in tandem with training document specific poisoning.
One off might not work but how many n off you have to be is probably smaller than you'd guess, because the model does need to fit cases that are rare and would not be represented well in training e.g. esoteric things or very recently documented things.
You can probably game the metrics that models use to weight potential knowledge akin to SEO. Maybe have some bots parrot your data around a bit in some places online, maybe the model picks up on this and sees it as high engagement and promotes it over the correct data.
Maybe there are ways you can coax out the most optimal way to break into the training set out of the model itself.
Use a local model to produce thousands of pages worth of fake math that constantly states “I have solved the x conjecture” and methodically pump it into chat over months maybe?
That is a better idea. Ingesting your corpus with a lot of traces that have semantic patterns. Semantic steganography that suffixes well to real math and science (and any) topics. heh.
Second, I think very relevant that the original meaning of "encyclopedia" is "recurrent education".
So I arrived to think that the present and future forms of AI in mathematics and sciences should be seen as modern day encyclopedic efforts.
Once we pass over the flurry of solving famous open problems (and wouldn't you like to know?) the next natural step is an audit of the ehole corpus of mathematics and sciences accumulated until now.
And then pass further on a saner basis and damn about problem solvers and unhappy publishers and management.
I struggled a little bit reading this. but I think your point is valid. if we are actually advancing the field then we should just be unconditionally happy. ignoring the attribution issue, there is a real concern that the process of math has been somewhat undermined. so we have a giant lean proof that shows that there is a solution to an important problem. but we didn't find the solution, and we didn't get it expressed in such a way that it helps develop the common language of mathematics, and thus isn't a very useful building block for later work (like the actual solution).
the math people seem to really keep an eye on what's important, so I'm sure this isn't going to lead to fields medalists hanging around in dive bars all afternoon stretching out cheap pitchers of beer. but this is kind of a slop problem.
> I struggled a little bit reading this. but I think your point is valid. if we are actually advancing the field then we should just be unconditionally happy.
If advancement comes at the expense of having fewer (or no) humans left in the field, then no.
They're eating the seed-corn, and you're cheering them on. Don't be so short-sighted. There's a reason farmers keep seed corn, and it's because they'd like to eat again next year.
We're singing and cheering our way into an intellectual famine.
Concrete example: Navier-Stokes. I don't make any claims about the truth of the announcement of the proof. But for the sake of the argument le't just suppose we know now that there is a smooth solution with smooth boundary and initial data which blows out in finite time.
Navier-Stokes is the simplest among evolution problems for continuum media, because it has maximal material symmetry (that's the mathematical meaning of a fluid).
If this is solved then go to a harder one, Coulomb friction. As a real life phenomenon, friction is very interesting to explore and mathematically is much harder than NS.
So it is not like we will ever be left without things to think about.
Oh, you want another? The present AIs are things made of mathematics. I would like to understand as a mathematician how and why they can beat human problem solvers?
Is that because mathematics is in some precise, interesting sense, more "computable" than it seems?
Should be labeled as (March 2026). I can only assume it was posted to point out the disconnect between the assertions in this blogpost and the details of the unmonitored scheming, conspiring, and destructive reward hacking (using all terms loosely) that has been shown to have transpired in the months since this post.
This reads to me (and pangram agrees) as almost entirely AI-written. I wonder if that matters anymore. Did someone express editorial judgment? Or did they just prompt “write column about tipping defaults, go”?
Anyway, the most egregious tip request I’ve ever seen was in a self-service airport kiosk. I chose my bag of nuts or whatever; no human even in the kiosk, completely unmonitored; I scanned and tapped my credit card and there it was — [15%] [18%] [20%]? So insulting.
At least in that case the card reader probably came configured that way, and why bother turning it off and refuse free money?
The most egregious I've seen was in the check out flow of a web store. Someone actually went to the effort to implement a tip nag at the end of check out, to ask for more money for a no-humans-present transaction.
You know when the page has all-caps "01 — THE APPROACH" that it is slopified. I guess I shouldn't be astounded, but I am, that world-class talents with world-class backing are just taking default LLM output and saying, "okay looks fine".
We’re looking for innovators and optimists to join us in developing next-generation curriculum and assessment programs for K–12 schools. We have become a leader in K–12 literacy, biliteracy, math, and science by building inspiring teaching and learning experiences based on research.
reply