Neither are compilers? But you still trust your JIT compiler. Determinism is unimportant, likelihood of correctness is what matters. Plus agents are deterministic, no?
Even if we accept your premise that the likelihood of correctness of more important I still don't think that the two scenarios are similar. I have seen more incorrect outputs from LLMs in the past week than I have seen incorrect outputs due to compiler bugs in my entire career. And without determinism there is no way to be sure that ratio will ever change.
I'm not sure. That is also my experience, but I'm not completely sure that isn't my fault and the compiler comparison would be totally valid if I just used the tool better. When I actually write fully fleshed out specifications and have the LLM handle the implementation, I rarely encounter bugs.
I don't think I've seen a Sol 5.6 or later model fail at a task with a verifiable outcome.
It's when I try to do too little of the defining that I get LLM bugs. In which case maybe programming hasn't changed that much, LLMs really can be reliable compilers, and we're all just being lazy.
Except they don’t right? They aren’t deterministic. All the hypemaxxers over at twitter keep saying you need a way for the agent to verify the result itself. Would that be another prompt then?
Cannot fail? It's so smart it literally can't ever parse incoming content incorrectly and provide the wrong answer to a question like "Is at least 25% of the text content in this document written in German? (yes/no)"
You are talking about the content of the response, while krial is talking about deviations in the structure of the response. The point is the number of possible states an output can take, so the next step can ingest it and it can be part of a robust pipeline.
https://x.com/LiveOverflow/status/2114412072805208237