My first deep dive with Claude Code was in Jan of this year. I was working on rebuilding an old city builder game from the the 00s and was trying to parse the original images from the game files. There was plenty if info on the structure online so I said I'd let Claude give it a go. I was running one of the sonnets I think.
After about 5 prompts I had a web interface that I could drag the files into and it presented me a grid of all the images and I was able to export more common sprite sheets. Whole thing took maybe 10-15min and my mind was blown.
I have tried to do the same recently with the newer models and we're still not writing code by the time the older models had finished their work. Something has definetly changed in relation to how much pre work the newer models are doing and maybe it's by design for more consistant results.
Not commenting on speed, but it seems models and their harnesses have generally gotten much worse over time. My suspicion is that the "Frontier" labs really dont have a strong handle on good quality evals that equal expectations of their users, so they just churn out new models for marketing to sell the shit out of.
Recently cancelled my Codex subscription because I cannot stand 5.6-sol/terra/luna. On top of that, Codex the harness is just so dammed buggy in VS Code.
I just discovered https://tinfoil.sh, which is fully private and attestable AI which works amazing with Kilo Code. Its costing me at API prices but for cheaper to run models which so far feel more consistent then what I get from frontier models.
As someone working in this space: unless they're a very big player with resources to create custom hardware, afaik there's no way to actually prevent the GPU host from seeing the request content. There's no "secure enclave" that sits between a CPU and the GPU, the decrypted payload must necessarily hit system RAM, as would the output. They are, at best, running in TEE VMs which are also pretty horribly broken and don't provide the security that they say they do.
Specifically, the intel page says "Intel® Xeon® 6 processors with Performance‑cores support Intel® TDX Connect technology, enabling confidential computing across the CPU and connected devices including GPUs, Smart NICs, and storage." which you are claiming do not exist....
Not trying to be an evangelist for them, but they seem to pretty open about their tech stack which is hugely differentiating compared to every other AI company....
> I have tried to do the same recently with the newer models and we're still not writing code by the time the older models had finished their work
Claude used to run at ~150tps in January. Now it runs at 50tps if you're lucky. That explains most of the speed issues without even getting into technical details about the new models.
Ya, it means my role at Google is gone and I have to find a new role or be laid off. I think “on the beach” sounds better than gardening leave, the term comes from the American consulting firms.
That’s weird. Isn’t that kind of on the employer to decide whether they have different work for you? Are you expected to be actively doing something during this period?
No, it is exactly like gardening leave, except you have a chance of switching to a new role at the company by applying for internal job transfers. I'm abusing the term also, for American consulting firms "on the beach" means they don't have a new client lined up for you and you are in danger of losing your job (vs. needing to do an internal transfer). Gardening leave I think is more of a UK term that they use when they want to lay you off but have to pay your salary for a certain amount of time before then, but its been a while since I heard that term and I don't want to cheat using Gemini.
Edit: ok I cheated using Gemini. Gardening leave is you are leaving or were terminated, and they have to pay your salary but don't want you near company assets. On the Beach is you are waiting for your next assignment and have access to training and such, you aren't consider a security breach.
I'm in both worlds right now: I wasn't technically terminated, so I can look for a new position (like being on the beach), BUT they think I'm a potential security breach so my access to corporate resources is really nerfed (like on gardening leave).
Gemini calls it being a "displaced employee". I like to think I'm just on the beach in Bali working on my own projects.
I’ve heard the term “on the bench”, which means you have no project to work on and you are lined up for firing if the situation doesn’t change soon enough.
After about 5 prompts I had a web interface that I could drag the files into and it presented me a grid of all the images and I was able to export more common sprite sheets. Whole thing took maybe 10-15min and my mind was blown.
I have tried to do the same recently with the newer models and we're still not writing code by the time the older models had finished their work. Something has definetly changed in relation to how much pre work the newer models are doing and maybe it's by design for more consistant results.