Last year, the frontier llms could produce working code, but the code looked brittle and overly defensive. Earlier this year, it got better and started to take more of the codebase into account. Since Claude 5.5, the quality improved substantially.
A lot of these things could be improved with prompt engineering and steering, but much less so nowadays.
But I don't think code quality is easy to measure, so the best we have is people's experience.
I've used local models extensively (mostly qwen 3.8 27b) and now Claude sonnet/opus 5.5 since launch.
Sometimes I really wonder if not being able to see what's going on in thinking is a large contributor to thinking the models are stronger.
That assumes you bother skimming through thinking traces and the output in-between.
In my experience both qwen and claude fumbles and misunderstand, but will in a lot of cases self correct.
Qwen models can feel very indecisive but somehow makes it in the end. If the harness was able to hide that and only highlight the end results, I'd probably think qwen models are smarter.
Reading the tool calls Claude does while thinking shows a lot of trial and error about even simple things like editing code and getting the whitespace wrong. But it happens so fast compared to qwen that it's sort of acceptable. With qwen, it can feel frustrating because it spends so much time on it.
This is all assuming that's what you meant by mistakes, but maybe you meant something different.
In my experience, the science/math/code crowd don't care about copyright/plagiarism as much as the video/art/literature crowd, so the first crowd turns a blind eye to most of the latter crowd talks about.
Code can be art, and copyright/plagiarism is real. It sort of boils down to how much it bothers us.
Sure, but I'm an atheist. I got to being an atheist by growing up in a religious community that I ultimately rejected. But I understand the experience of that which people crave and I understand why they feel sorry for atheists.
I'm not saying a soul is given by God or that one needs to believe anything in order to find a soul in oneself. What I'm saying is that the poster's comment is darkness and hopelessness and a hatred for his own species rolled up into one, and that such a perspective is both painfully inexperienced and profoundly awful for the person and for whoever they are convincing or it, because life contains much more richness (religious or secular) than what is disdained by this person.
What you find condescending in my statement isn't that it's religious, it's that I'm willing to tell them that they're missing out on life. In that respect, I don't mind being condescending, because not all philosophies or ways of living lead to equally happy results.
I would say opus was in some ways more accurate, but missed the higher level curvature feel of the site that astra picked up on.
It sounds completely trivial and likely I'm wrong here, but could it be that opus saw the reference image squished? That might explain the sharper horizontal curvature
I've been vary of using ai to search considering all the spam out there. I think I'd rather, perhaps naively, whitelist wikipedia, reddit, arxiv, some news sources, etc than include everything.
Is there nothing out there that does this? I'm paying for kagi and I can see that it has an api, is that maybe sufficient if configured properly?
If you use Kagi Assistant, you can pick one of your lenses (i.e. lists of domains to restrict searches to) in chats. Not sure if their API has that as well or some other way to restrict searches. Also not sure if the Assistant (or API) respects blocked domains when searching.
I direct them to search for discussion on fora. There are still legacy sites on niche topics that the bots don't post in. This is particulaly useful for reaching into the past because traditional search doesn't surface anything but new content.
ThinkingCap is a 3.6 27b finetune that claims to halve thinking tokens while maintaining the same output quality. I've used the model a lot and I'd say it holds up. Since 3.6 has the same architecture as 3.8, the lora can be applied.
With the prompt "create a fancy circle in html", these are the results for xhigh, medium, low and xhigh + thinkingcap lora
(Note that the gists are bloated because they contain the full chat and launch params in text/plain script tags for transparency)
I'd say xhigh looks a little better than xhigh + lora, but the lora variant has 40% less thinking tokens. Both seemed to take the same approach with adding random details that weren't explicitly specified.
Medium and low (no lora) are close to each other but are much simpler results.
This is just me testing a single turn. I haven't tested this on multi turns and whatnot, but I thought the result was interesting enough to share anyway.
Scale 35 felt a bit noisy and incoherent, but 30 seemed nice. (they use the same seed, but idk how reliable seed in llamacpp is)
I use a python test script that captures the answer and renders it to a html page along with the llama-cli log, launch parameters, the chat log, and the python script itself for maximum transparency. :)
Last year, the frontier llms could produce working code, but the code looked brittle and overly defensive. Earlier this year, it got better and started to take more of the codebase into account. Since Claude 5.5, the quality improved substantially.
A lot of these things could be improved with prompt engineering and steering, but much less so nowadays.
But I don't think code quality is easy to measure, so the best we have is people's experience.
reply