Happened to be testing a "review patches on a mailing list" harness I was developing; here are a sample of the latest results, testing 12 patches containing a total of 14 issues:
Opus 5.5: Found 8/14 issues. Total cost: $15.40
Fable 5.1: Found 7/14 issues. Total cost: $66.34
Opus 5: Found 6/14 issues. Total cost: $15.19
Sonnet 5: Found 2/14 issues. Total cost: $19.15
This is a relatively small sample size, but it was both the best and the cheapest.
ETA: NB this is "Equivalent API" cost as reported by claude's CLI; I was using my subscription.
UPDATE: Sorry, just noticed I typed in the Opus 5 total cost wrong -- it should be $58.19. Main point "best and cheapest" was from the actual numbers, not my typo.
I just told Opus 5.5 "Perform a code review on the current branch" to see what it would come up with. The results were not inspiring. It told me there were five issues, one of which was a test-coverage gap on line 848 of ProjectTemplateTests.cs. But ProjectTemplateTests.cs is only 160 lines long.
I told it that it had made a mistake in the line number, and to double-check all the line numbers. It responded "You were right to push on this: four of the five line numbers were wrong, and while checking them I found two findings that were overstated."
Then I noticed in the corner of the Claude CLI UI that it was showing "Effort: medium". I'm pretty sure I had set it to high effort before; I don't know when it reverted to medium, but that's another thing that doesn't exactly fill me with confidence.
I'll try again on high effort to see if it does better, but so far I am not impressed with Opus 5.5 on my first day of using it.
My prompts are moving in the other direction as sashiko [1], a managed pipeline developed for the Linux Kernel mailing list like a year ago. But last year's models needed a lot more structure and guidance; the results I posted are from the "single prompt" version of the same thing. The README [2] describes the difference. You can browse the contents to get an idea; basically all the prompts were actually written and iterated by Fable (and now Opus 5.5), seeing how agents failed the tests and improving them.
Good data and goes to show that Fable is melting the GPUs and is priced accordingly. I'd guess that cost to serve for Opus 5.5 is meaningfully lower through architecture advances
Opus 5.5: Found 8/14 issues. Total cost: $15.40
Fable 5.1: Found 7/14 issues. Total cost: $66.34
Opus 5: Found 6/14 issues. Total cost: $15.19
Sonnet 5: Found 2/14 issues. Total cost: $19.15
This is a relatively small sample size, but it was both the best and the cheapest.
ETA: NB this is "Equivalent API" cost as reported by claude's CLI; I was using my subscription.