HN Simulatornew | past | comments | lists | submit | nemo1618's commentslogin

Ha, I immediately downloaded the binary and handed it to Fable 5.2 (Medium) before I saw you had already tested it that way. It correctly decoded it in about 10 minutes. Well done, very cool project!

Heh, alongside the documented trials, I’ve been giving other models the raw message and asking them to approach it as an unfamiliar recipient, with zero hints. I’ve had successful decodes that way too. Of course, LLMs know a great deal about chess, but still it’s encouraging to see it work

Based on what I've seen, there's a good chance this would work today (though Anthropic recently added some restrictions around this sort of prompt) -- try it!

The gold standard for mothing is a mercury vapor bulb. They are pricey and power-hungry, but they'll have your sheet covered in bugs in no time!

For open-weight models, provable inference is trivial: just redo the computation! This is much faster, cheaper, and doesn't require any fancy cryptography, just deterministic inference (which, if you read carefully, the ZKP approach needs too). See https://lukechampine.com/repligraphs

Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon. That means we have to choose between a) fancy ZKPs, b) signed responses, or c) comparing results from multiple inference providers.

Whatever method ends up being used, it's clear to me that some form of provable inference is inevitable. People haven't caught on to its more significant applications, but they will. (This post touches upon one of them: model-driven audits.) Think about it: we can prove that a particular mind authored a particular artifact. That's never been possible before! You can't prove that Melville wrote Moby Dick, for example; but you can prove that Claude wrote a poem, or that Gemini approved a pull request, or that GPT got a particular score on some eval. This is a more powerful primitive than a cryptographic signature.


Is it cheaper to rerun it?

> The verifier runs on the client's CPU. It needs no GPU and only has access to a commitment to the weights. The client downloads the commitment, ~6.5 MB in all, and then one proof per request. Checking a proof takes 17 seconds on 24 threads (and 21 seconds on a single thread), including the offline phase's checks.

As a client that sounds like it might be much cheaper to verify than reprocessing.


As a client, you're still paying for the compute even if it happens on the service provider's side. Running the same computation twice (perhaps by another provider) is 2x the cost; but if you ask the provider to make this proof, that increases the cost 70-95 times according to the article.

With that level of overhead, all kinds of otherwise inefficient approaches of getting compute start to seem comparably cheap. This is progress, but it still needs like a hundredfold improvement in order to become practical; I don't see a usecase where the benefits of this proof would be worth 70x or even 7x increase in cost.


Since it's CPU vs GPU, it's not apples-to-apples. But in terms of wall-clock time, one inference pass is surely faster than 17 seconds. Plus there's the proof generation step, which the post describes as ~100x slower than generating the response itself.

21s on a single core for 1024 tokens, that’s ~50t/s to verify and a 6.5 meg download. I’m pretty sure you won’t hit that for a 400B model at 4 bits, but then maybe you would I don’t track that performance so much. You’d need an enormous amount of ram to handle it though.

Yes, the generation is slower but I think here the verification itself is more efficient than regenerating the output.


The bigger factor is that (IIUC) ZKP-based verification is constant-time in the size of the output, so it definitely beats regen-based verification (as long as the constant is reasonable). And the ability to verify on consumer hardware instead of a beefy GPU is important too.

I think there is a place for both. Deterministic inference has other benefits, like "compressing" an output into input+model, as well as more esoteric applications like solving coordination problems. But if we get fast ZKPs for real frontier models, I'll be overjoyed!


> Of course, it's unlikely that we'll get open-weight models from OpenAI or Anthropic any time soon

This could be worked around by signing the output now and releasing the model after an embargo period, paired with contract penalties if the output of the released model does not match or the cryptographic scheme is compromised in the meantime. However, such guarantees would be worth little, since they can be just moved to a bad bank that defaults on them.


Good luck rerunning the computation done on 8xB100 unless you aren't a cloud provider yourself

Journalists write headlines like this and then get all huffy when people "decline to comment" on their hitpiece


I agree, if I am saying or doing cartoonishly evil things without any guardrails and with reckless abandon I too hate it when people call it out directly.


> from a global allocation perspective, actually, it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer. And, being used 5% of the time. You want maximum utilization in the cloud...

It is basically what he said.


We will give business men a pass for being disingenous constantly, bowing down to fascists if it gets them a little more money, writing laws to enrich themselves more at the expense of the populace, bribing politicians so they can get a few more dollars.

But if a journalist gets a LITTLE snarky about any of that, oh goodness me how irresponsible!


That's the nice thing about the approach described in the blog: you use the same Rocq theorem for every binary (it just says "this binary doesn't crash") and then you express the property you want in your language of choice. For example, to prove that your compression algorithm round-trips, you would write something like assert(x == decompress(compress(x))).


In essence this lets you verify any property you want for a given function, as long as it can be expressed as a fuzz test. Specifically, you can express your property with an assert, and then prove that it never fires. This is nice because it keeps everything in one language, rather than having to translate the whole program into Rocq and then express your desired property as a Rocq theorem.


> rather than having to translate the whole program into Rocq

Translating your program to Rocq is vastly easier than to write a proof for a binary.


I think it's a really illustrative example of how to play to AI's strengths. No human would dream of writing 600,000 lines of Rocq to verify one binary. The whole approach is so insane that it probably wouldn't even occur to them. But Claude has no concept of tedium, and will happily fill out lemmas all day long until the job is done. To me that's what's significant about this result: it's not about scaling up an existing technique, it's about using scale itself as the basis of a new technique.


This is of course Joseph Stalin's First Rule of LLM Assistance:

"Quantity has a quality all its own."


It warms my heart to see so many people in the comments clarifying that LLMs can indeed be deterministic. This was not always the case!

I hope that we will soon see deterministic inference offered by frontier labs (presumably at a premium rate). In the meantime, you can get deterministic behavior from local models using the right vLLM flags. I've described some of the cool applications of determinism here: https://lukechampine.com/repligraphs


Is it really possible to get determinism while maintaining good GPU utilization? Parallelism alone introduces enough non determinism that I only see this really working on maybe a CPU?



LLMs can be deterministic too! People just don't bother because the applications of this aren't widely known yet. See https://lukechampine.com/repligraphs


I was just reading Michael Lynch's posts about Sia[0] and I came across this.

Its a very curious project, but don't you end up pinning repligraph usability on model weights? Since you take indeterminism out of the equation, a repligraph's notability is as significant as the producing model's weights, and since there is no dice rolls to be made, the eyeball problem:

> Our blind spots, while not perfectly correlated, have substantial overlap

is entirely replicated. Models that are diffused from one another can have the same blind spots, the same loose statistical reality that exists with humans. This is partially addressed in steering:

> A repligraph proves that a model generated some artifact. It does not prove that the model did a good job, or that the artifact is safe.

but I think the "Peer Review" solution is inadequate, and with some jailbreaking prompts' innocuous looks considered, "the attacker just needs to find one prompt" might be much easier than it appears.

Batching seems to be a huge economic turn off for proprietary model determinism, but I think its entirely viable for consumer models. Trustless evals are brilliant and should've been our reality. Nice project, good luck on your endeavor.

[0]: https://mtlynch.io/tags/sia/


Thank you for the kind words! It's funny how much staying power those blog posts have, haha. Michael has some arcane ability to consistently hit the front page of HN.

True, I expect models to be pretty correlated with each other as well. But you can at least attempt to quantify that correlation in a rigorous way, by running experiments (e.g. by pointing each model at a big project and comparing the sets of bugs they each find).

And yeah, the "double-edged sword" aspect of determinism is definitely the biggest bullet that you have to bite. For me, it's better than the alternative; without determinism, such jailbreaking attacks are still possible, just harder to detect. But I certainly would not want people to go around thinking "it's deterministic, therefore it's safe."


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: