If you're happy with your agent's performance using surface level o11y to track down issues then that's fine. This is what pretty much the entire industry is doing and it does usually work! QA focuses on root causes because it can look through the app's entire execution and provide evidence so your agent doesn't need to guess when figuring out what the fix is. That guesswork is what we're trying to eliminate, for when the surface level stuff doesn't clearly point at the problem.
What sort of bugs do you have in mind? From our perspective there are many kinds of bugs that inhibit UX (and AX) but they can be subtle issues like poorly designed flows. It's hard for an agent to notice or understand these but they are serious problems and we're trying to surface these more readily.
Agents that are blind to understanding the problem they created will definitely not be able to fix it. Our argument is that we can fix that blindness with better observability. For better or for worse, even if the code is messy, if an agent can see what is going on it can keep things together and get the app into working shape. 100% human written too btw...
I think AX is just as important in that case though, as even a patient agent will make mistakes and will have a lot more trouble getting things done right when using a buggy app.
There's an opportunity here to bend things towards quality and efficiency moreso than mere volume of code. The agent has to be able to effectively repair and maintain what it has been building though.
(OP) I completely agree! AI is amazing at writing web apps that are full of bugs, but it also gives us what we need to turn that completely around. That's what we're trying to make happen...
>> It is possible now to reliably and automatically detect and fix every bug
which users are encountering on the web.
proceeds to give a trivial and then vague example related to usability, the first probably created by AI and the second a notoriously difficult problem to assess and solve for humans with context, let alone AIs...
The code I have dealt with from AI tends to be rather convoluted for the problems at hand and then become even more difficult to maintain. It has a place but I have been seeing it make code more difficult to understand, not less.
Convoluted code is definitely a problem, I don't think it's unique to AI though. It also becomes more manageable with a time travel debugger as the actual behavior is easy to understand.
Yep. I see it myself, both in other people's code, and our own. The _default_ behavior is a tendency to pile slop on slop, make poor architectural choices, slap bandaids on edge cases without considering the full ramifications, etc. This is more true if there isn't oversight or feedback, and also depending on model quality.
On the flip side: I personally use a strongly human-in-the-loop AI dev workflow [0]. Going back and forth on how much of that code I review myself (and I've found that AI code review frankly does a better job than _I_ do most of the time :) ). But, between staying hands-on, and the ongoing improvements in models, I've seen that the code my own agentic dev sessions produces is actually pretty high quality, and I've been able to clean up a lot of low-quality slop from unmonitored agent sessions. I've also been able to build some pretty major features from scratch and even execute massive migrations with excellent results, again by having me drive the work.
I actually just got done shipping Redux Toolkit 2.14 [1] with a massive set of bug fixes, and almost all of those were found by my agent researching the already-mature RTK codebase, at my direction, and identifying existing logic flaws or weaknesses. Once we knew what those bugs were, we could fix them.
Which, honestly, is the point of the blog post, of Replay QA as an automated testing product, and of Replay as a company in the first place.
Once you know what a bug _is_ and why it happens, fixing it is usually fairly straightforward. _Finding_ it is the hard part.
I've given a conf talk the last few years on "Debugging Software" [2]. One of the main points I make is that no one is _taught_ how to debug - we figure it out on the job as we go, because we have to ship. It's part of the job. But few people like doing it, and most people just want to slap in some console logs and call it a day. Very few people even use existing graphical debuggers (VS Code, Chrome DevTools, etc).
We built a time-travel debugger. The original idea was: make a DVR-style recording of a bug happening once, then you can dive in as deep as you need to investigate. Originally, that was supposed to be a human using the Replay DevTools UI - browser devtools with time travel debugging features built in. It's awesome, I wish I'd had it much earlier in my career :) But we found most people didn't want to spend time doing it themselves.
Earlier this year we shipped Replay MCP, which gives agents the same time-travel debugging capabilities: see hit counts for any line of code, dynamic logpoints and evaluations for every time a line of code ran, see the DOM tree or React component tree at any point in time. I built most of the React analysis features and MCP tools myself, based on my years of experience in the React ecosystem and maintaining Redux.
So, today you _could_ make a Replay recording of your app, point your agent at the MCP, and have it investigate. I've done that, and wrote some blog posts about how that helped my own agent find bugs in React itself and Next.js apps [3] [4]. I've seen my agent nail a bug from a recording on someone else's machine in <10min, starting from scratch.
We've productized that as Replay QA, so that you don't have to do any of the recording or setup - just point Replay QA at your app and repo and we do all the site analysis and recording and bug root cause investigation ourselves. Essentially, encoding our own expertise as debuggers, so you get the benefits [5]
So, back to the original point. Yeah, AI tends to crank out bad code by default, that code is buggy. We can't help your agent choose a better architecture. But we can help identify the bugs proactively, identify the root causes, and suggest solutions, based on the actual runtime behavior of the app. Up to you to get your agents to write _better_ code, but we can give your agent what it needs to fix the bugs it wrote :)
reply