What is a better analogy for how you think developer tools should think about agent experience: an RL environment (hillclimbing a task like you mention), or a test suite (to be used in CI/CD, etc) as referenced by Supabase/Vercel?
Active work should mostly be hillclimbing, or at least more exploratory. The search space is very large and you typically find a lot more low-hanging friction than you'd think by just running a broad battery of sessions. And I'd pair active hillclimbing work with feature launches/product work - essentially part of releasing new things is QA-ing them against coding agent workflows.
The test suite eval style is useful and mostly a complement. You do want regression tests on critical features.
The search space being large feels like a fundamentally hard and costly problem to solve for. Where do you recommend folks start given such a high number of possible scenarios?
How do you think about enabling fully agentic onboarding without proper human identification? It feels like any rate limiting would be pretty easy to get around.
Each product is different (how much does it cost to serve a marginal user, how risky is it). But generally I think allowing ephemeral accounts like Cloudflare, or serving simulated/static accounts before a human claim can both work.
For a lot of products it can just be an extension of the free tier I think too.
- https://supabase.com/evals - https://nextjs.org/evals
What is a better analogy for how you think developer tools should think about agent experience: an RL environment (hillclimbing a task like you mention), or a test suite (to be used in CI/CD, etc) as referenced by Supabase/Vercel?
reply