I'm trying to fix the "agent said work was done, it was not" problem by defining work as runnable specifications in the form of a pytest suite.
Work is first defined in new or updated specifications, then the change gets made. You have to resist making natural language Todo lists and have the agent write runnable unit tests. My project is a CLI tool so it's fairly easy.
Verifying if it's done is a matter of running the specifications test suite.
In a totally not original way, I'm testing this workflow in my own agent orchestration tool. I know, there's just so many already. I'm building it for myself.
So far, the system is holding up but it's way too early to declare it a success.
I think the same idea could be applied to other projects in different domains.
With this workflow I can read and update the specs, and run it against a built binary and be much more confident that the code works. Claude code has picked up the system without complaining for the most part.
Work is first defined in new or updated specifications, then the change gets made. You have to resist making natural language Todo lists and have the agent write runnable unit tests. My project is a CLI tool so it's fairly easy.
Verifying if it's done is a matter of running the specifications test suite.
In a totally not original way, I'm testing this workflow in my own agent orchestration tool. I know, there's just so many already. I'm building it for myself.
So far, the system is holding up but it's way too early to declare it a success.
You can check it out here:
https://github.com/egzo-ai/egzo/tree/main/specs
I think the same idea could be applied to other projects in different domains.
With this workflow I can read and update the specs, and run it against a built binary and be much more confident that the code works. Claude code has picked up the system without complaining for the most part.