Does your research able to take most relevant stuff? Let's say i search for flight tickets, is it able to really find the most cheat ones

Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on.

I’ve tried it on simple UI tasks. Give it screenshot to work towards (or figma MCP), let it get screenshots from chrome to check its work.

The existing models are surprisingly bad at it.

It's really difficult to understand what your definition of surprisingly bad is. What was it continuously having problems with?

Just reporting work as done that’s not even close mainly. Then if you manually screenshot both and ask if they are the same it’s like “yeah looks great boss”. The visual understanding of the models is just leagues worse than their text/language understanding.

Or guessing colors rather than sampling from the image or pulling from figma is another stupid thing they do constantly.

I imagine Gemini would do better on these types of tasks. Curious to try it out.