I’m running little offices of 5-10 persistent agents with defined roles and it’s crazy productive and code quality has never been better. I’ve been working on management of physical systems as well, and so far it’s been pretty solid.
Do you basically know that quality of code hasn't declined based on the number of incidents that are occurring? How do you even know what is happening?
Every bit of code gets 10-50 review passes against extensive smells, rules, best practices, etc. Naive reviews and documentation driven development is your friend. If your code isn’t 100percent testable (even code previously considered untestable, because now you can create simulators and jigs) and you aren’t spending 5x+ tokens on review over coding, you’re going to have a bad time.
It’s still 5-10x less expensive than people, and you end up with higher quality code in the end.
I'd instead say that for both it's really at 100%, and not any number in between.
Problem is that we can't define precisely what is good enough or when software is "finished": I mean, we are trying to do that with human languages, so it should not be a surprise.
Yes, LLMs are now similarly aware of the average context a human would be aware of, but for anything specific to the situation a human has better chances of resolving the ambiguity.
Is there any great documentation you have followed to build your workflow or has it been experimentation? I've been experimenting a lot and gotten some cool things done but every time I read up on what everyone else is doing I realize I don't have much imagination and so I miss out.
I’m afraid I just grew my process organically, starting with cutting and pasting code into chat lol. But now it basically runs on its own, and I check my issue board every few hours for judgement calls on ambiguous decisions or redirecting in the rare cases that even bringing in independent reviews from other models can’t solve a problem. You have to have mechanisms to stop thrashing, or you’ll end up throwing away a lot of tokens. I use the second attempt rule. If something can’t be solved in the second attempt (which included at least one naive review) then they call in the gatekeeper agents that gate merges and commits based on code quality and standards compliance for a pow-wow. If that doesn’t result in a solid resolution or there is a structural problem that cannot be resolved without major rewriting of other code, then it gets put on my board.
It’s basically an office of persistent agents with different defined roles and goals. The gatekeepers whole job is to refuse code. The engineers whole job is to write code that will pass. Each significant component gets its own engineer agent. There is an office manager that coordinates the agents, maintains the project documentation, keeps up the job boards, maintains scripts to keep agents pointed at the relevant onboarding documents As they come back from compaction, and manages office automation in general. Ther is an agent in charge of making sure the test coverage is as close to fully tested as possible, including refusing code that is not properly testable.
If it’s a big project, there may be an information management agent that manages the technical side of the documentation and maintains a searchable library, like an oracle. I may put in a personal assistant to help me to prioritize my involvement. Many projects also include an agent that creates mockups, jigs, and simulators so that tests can be run on code that needs a context.
I maintain another office just for research, because research can seriously sidetrack engineering. It has all of the standard staffing, but its whole job is to do research and write reports for other teams.
Agent persistence is maintained by having agents write down memories learnings and musings(not work related) and curiosities for the next instance to read in the onboarding for their seat.
Ethos, standards, practices, and policies are also in the onboarding process.
