It's been the same for every version the last year.
Oh, this Opus X.Y is incredible.
Hmm... Now it's dumb.
Rinse, repeat.
Doesn't really matter if the cause is quantization, system prompt changes, changes to thinking etc.
It's hard to see how continuously rug pulling users at every release is a good idea. Only reason I can fathom is that it's needed to keep costs inline.
Either that or it's some weird tweaking to the prompts to fix whatever. But given the prevalent sentiment that it's worse that sounds like a long shot
I also use global ignore, but I think the key is to keep secrets out of commands. I store mine in .dev.vars, which Wrangler reads automatically. For production, I use wrangler secret put and enter them myself. I also told Claude to never read that file.
There's pretty much always several plausible reasons for why a change was made. What's interesting is knowing exactly which one, especially when it later turns out to be wrong!
Claude tends to just narrate the change when it writes the commit message. Which is not very interesting. Anyone can read the diff and figure out _what_ it does. The interesting is why.
So I've been instructing Claude to commit like Jeff King.
At first, Claude would mainly just cosplay Peff. Emulate the prose and not the process. Over the last few months I've been iterating on it and now Claude writes vastly better commit message than by default.
Initially, Claude would produce A LOT of plausible sounding reasons the LLM "thought" made sense. Instruct an LLM to give reason and it'll give you reasons -- whether they are real or not. After trying to instruct it not to lie, make shit up etc (which did not work) I instead started forcing it to articulate the source of the rationales. Especially which claims where unsubstantiated, and this seems to have helped a lot.
Then I instructed it to do some thorough investigation before it commits.
Start by writing a brief that gathers different "evidence" that underpins a change. The diff itself. The surrounding context. A bit short git log. A blame on the touched lines to see what previous commits touched this code and for what reasons.
Once it's done the agent has to tag each claim according to a category. I.e. what claims are attributed to the change itself (the diff), the inciting incident (gathered from session or if missing, by follow-up questions), what's inferred by the model (unsubstantiated claims.)
Only after this supersize is it tasked with writing a commit message given this brief. Or to ask follow-up questions if there's only unsubstantiated claims or gaps in the brief. Furthermore, it is tasked with writing a note to detail assumptions it has made and, or other relevant bits of information that are not commit message worthy, but possibly still interested in noting down. Decisions made. Options not taken. Possible rationales for the change that didn't make the cut.
All of this tends to make pretty good commit messages. Not perfect, but a good starting point.
Right now my biggest challenge is finding instructions to write the Goldilocks message. Not too brief and not too long. Instruct it to be clear and concise and relevant information gets left out. Say nothing and get a Dostoevsky novel. At least when it writes too long messages it's easy enough to go in afterwards with a `git history reword` and take out the axe.
One of the biggest upsides has been, just as when you read a human that writes commit messages like this, is spotting misunderstandings. Several times I've spotted gaps in the reasoning of the message that doesn't match reality, and caught mistakes. A bit like when you use plan mode.
Claude has been so bad that I find it easier to hand write commit messages and PR description. It works much better than nudging Claude to write things down in a readable and meaningful way.
Latency. My home setup has a different number of screens vs the single virtual display on the headless workstation/instance. I don't want to deal with a Linux GUI shoved into my Mac via screenshare. Linux screenshare host is finicky in the first place. I do lots of stuff via CLIs anyway. I connect to multiple remote instances at once. All those reasons make it easier to SSH.
My theory is it writes these comments as "notes while working", and I don't don't mind that, per-se. My problem is it's inability to clean that shit up before committing it. That's wrong load-bearing lever that doesn't earn it's keep.
Oh, this Opus X.Y is incredible. Hmm... Now it's dumb. Rinse, repeat.
Doesn't really matter if the cause is quantization, system prompt changes, changes to thinking etc.
It's hard to see how continuously rug pulling users at every release is a good idea. Only reason I can fathom is that it's needed to keep costs inline.
Either that or it's some weird tweaking to the prompts to fix whatever. But given the prevalent sentiment that it's worse that sounds like a long shot
reply