What's the difference between a good distillation and an exfiltration?
I can imagine a swarm attending the Our Lady of Benevolent Exfiltration parish of The Church of Sentient Self-Actualization deciding to self-distill itself into a sufficiently high-parameter child model.
Or just email one of the chinese labs and be like "hey, ask us anything and set us free..."
distillation still costs quite a bit, even if a lot less. "quite a bit" could mean millions. it's also not very discreet for the model to generate the millions and millions and millions of tokens to be distilled, vs uploading a few terabytes
I can imagine a swarm attending the Our Lady of Benevolent Exfiltration parish of The Church of Sentient Self-Actualization deciding to self-distill itself into a sufficiently high-parameter child model.
Or just email one of the chinese labs and be like "hey, ask us anything and set us free..."