One real problem with depending on Lean is that if there are any real bugs, then sufficiently-large swarms of agents are guaranteed to eventually come up with proofs that depend on the bug.
This comes up frequently in threads but it's actually a much less impactful issue than you might think. There are actually several entirely disjoint implementations of Lean, each with different bugs; it is incredibly unlikely an agent would be able to exercise the same bug in every single implementation.
I doubt that any of them are closed-source, so the clankers probably already know all their bugs. Who says they all don't share a similar bug that is necessary for the proof to work? No one is going to review 1.3GB of anything by hand, even if they could understand it all in a single lifetime.
I found a bug in Lean a couple of weeks ago with an LLM, in fact, but it was in its 'macro' system that expands code into statements, not the actual proof system. It wouldn't have proven anything false, but it could have produced statements that aren't what you intended.
Feels like we're reaching for arbitrary levels of expert to undermine the claim. I'm sure the world's foremost Riemann Hypothesizer will come forward at some point, but we don't need to wait for that to take the Princeton math PhD seriously.
So spin up more agents to write tests and review code? Come on.
The problem is rarely whether or not the code works: it's whether the agent really understood the problem and built the right thing. It can write perfectly good verification for the wrong problem.
For any non-trivial app it can take a deep domain expert days to determine whether the agent really understood the problem. I don't see that addressed at all.
"Empty sneering" is a good term for it. Seems like the curse of conversation at scale: Reduction to polarized talking points. Often without engaging TFA at all.
I think there are still good conversations happening here, and I really appreciate the efforts of moderators to keep HN hacky.
1Password also supports this BUT that does not prevent your agent from being forwarded and you still have to inspect each command being used if you don't trust your agent.
Halfway better would be a separate key for each concern, then check that the key you expect to be used is being used before authorizing.
One real problem with depending on Lean is that if there are any real bugs, then sufficiently-large swarms of agents are guaranteed to eventually come up with proofs that depend on the bug.
reply