100%, back when it was Ox Alpha I had a little fun trying to guess what it might be by looking at the reasoning and I consistently laughed at how excited it got
You are very much not alone, and I don't think we're all getting dumber -- I kept using opus 4.5 all through the nonsense that was 4.7, 4.8, and 5, and kept having a good time :)
I think we just need to decouple "doing better on benchmarks" and "actually more useful to me", since they've clearly diverged
That's not the point, the point is that the company making the product is optimizing for the benchmark and/or the apparently idiosyncratic preferences of their own team, and not for the user experience of their paying customers.
Company can optimise for the benchmark (profit) while worsening the product. I think thebterm enshitification is used there. It appears that AI got it too
> I think we just need to decouple "doing better on benchmarks" and "actually more useful to me", since they've clearly diverged
There's a third perspective here: models are getting less useful, but overfitting to seeming useful to humans.
Imho, this is why analysis like TFA + third party cross-compatible harnesses (read: last mile UX) are so important to the leading labs optimizing for actual utility.
I'm suspicious enough of my subjective evaluation to believe a well-designed harness / verbiage could gaslight me into believing an objectively inferior model was superior. And at some point frontier labs are looking at the ROI of investing $1 in that vs actual model improvement.
I'm unfamiliar with claudish and the example helped show the problem. But! There was something uncomfortably familiar in the Claudish example -- this is the way human programmers write when they're deep in the weedy details, and writing the changelist description afterwards as if coming up for air. Overuse of parentheses in nested lists especially, as if the English text needs to bend to the strict needs of a C++ parser.
The rewrite did seem to lose the important fact about the ensure- pattern being idempotent.
Indeed, probably not cheap. Ironically, they see themselves as the ones that should help others "stop overpaying for things" and "help companies make better spending decisions":
> Wait, Ramp bought router.com? We did. It was either this or spend the next year spelling out a longer URL on podcasts. More importantly, we build tools that help companies make better spending decisions and stop overpaying for things.
Indeed. Also, ramp.com is shorter than router.com, so I don't really understand the "spelling out a longer URL on podcasts". I guess router.com is just clearer for customers?
The "a3b" refers to its active parameters -- unlike 27b it is a mixture of experts model, so it runs much faster, about as fast as a 3b model, but needs as much memory as a 35b model! So good for unified memory systems like macs :)