What is are the benefits of supporting extremely high churn rate in version control? What is the code doing? What problems are being solved? Are changing requirements the driving factor, and if so, why are requirement changes accelerating?
> ..., the latent demand for merging code is practically infinite, and every bottleneck we remove immediately leads to a further increase in the amount of code we produce.
If the goal is to write software which solves real problems, how can the demand for code grow infinitely?
I think that they're essentially solving the wrong problem.
Imagine that you have git but you don't even have a concept of programming language more complex than assembly. All your commits and git history are pure assembly. Everything is cool, until one day C++20 randomly spawns into existence, and a bunch of guys decide to use it in the company. The issue is, the compiler generates so much assembly, that your existing workflow cannot keep up with it. Your assembly git repo grows into hundreds of gigabytes, with PRs that are pretty much complete rewrites coming every single day and there's no way to review any of that shit.
What Anthropic is doing is redoing git from scratch so that it can handle all that assembly, while the correct thing to do is to commit C++ code into the repo, with deep trust that you'll never hit a bug in the compiler - basically impossible for initial versions of compilers, taken for granted nowadays.
What I'm saying is, Anthropic should create a versioning system for prompts, because they are the true "source code". Given the same prompt and the same sequence of random numbers, LLM will generate exactly the same code, which sounds awfully close to how compilers work. Once in a while put the generated code as a point of reference+ caching and presto.
No one is actually one-shotting new code on top of the existing repo and just checking it in. Even in a simplistic case, it’s multiturn. You give it a prompt, and it writes some code, tries to build it, and it doesn’t work. So then it keeps iterating, and changes some more code until it builds and then does the same thing again until it passes tests. And if you have a complex system, it might do a lot more CI.
The input to this is not just the preexisting code, the prompt you provide, and some random seeds. It’s also the output from every single tool invoked, which means you need to record the state of every single tool and every file that was read, every API that was called. It means that you can’t upgrade Claude. It means that you can’t upgrade the compiler. If you change anything in the state of the world you no longer have replicability.
Aside from all that, I don’t know why you would want this anyway. Storing a bunch of prompts doesn’t seem to actually gain anything even if it worked.
> What I'm saying is, Anthropic should create a versioning system for prompts, because they are the true "source code". Given the same prompt and the same sequence of random numbers, LLM will generate exactly the same code, which sounds awfully close to how compilers work. Once in a while put the generated code as a point of reference+ caching and presto.
Bad analogy, cause this is not how tooling works. Tools beget more tools - there's value in specifying "these are the sets of tools we need to make", but unlocking new capabilities doesn't happen unless you... make the tools. And use them. And keep them stable. And watch them break, or become obsolete, and build on top of that. The world has state and path dependent.
Except they don’t right? They aren’t deterministic. All the hypemaxxers over at twitter keep saying you need a way for the agent to verify the result itself. Would that be another prompt then?
My thought too. Like, I get that Claude gets new features pretty frequently but.. what on earth are they possibly doing that needs that much churn?
The only thing I can think is that they're doing a google style mono-repo, which like, fine, that's a thing to do, but I really have trouble picturing that changing code at that rate is a useful thing to do.
If you are constantly churning your codebase to the point that you need to invent a system that handles your churn... then you must at least be admitting that the previous 6 refactors you wrote were not good, but NOW you have it right.
I feel that good, well designed code change by accretion, not by rewrites. Code that remains stable for a long period of time was likely better than code that needs be continuously rewritten (at least with humans writers).
But maybe this is just a human-centric view of what good code means? That LLM "slop it until it works" approach to development will have a different measure of what quality looks like?
idk, over the past 2 weekends i have a repo for hacking on fujifilm camera firmware that's gone from zero to tens of gigabytes: i've got an emulator for older arm cpus like the one in the X70 (2016), a Hypervisor.framework VM for running their newest AArch64 software for X100VI (2024), a complete C re-implementation of the x70 source, many tools for injecting code and executing on live camera RAM over USB. the emulator has gone through 3 rewrites: first it was python and naive, then it was python with hot parts in rust with a more mature device interface, and now it's rust with some cold or experimental devices still python.
i really don't want to brick my camera, so the deterministic emulator stack lets me do integration tests end-to-end. if i was doing this by hand with no ai, i'd probably have to run stuff on the camera just to test it. but because ai, i get deterministic simulation testing, at the cost of needing to burn $100 of subscription on it.
ai code is more like clay you gotta continuously keep shaping over time, but you can also get very very far, very quickly if you decide the code itself doesn't matter and all you care about is results.
Agreed, a buddy and I wrote a simple mod that shimed into Warhammer 40K: Rogue Trader and were able to pull all of the art and description assets for that game by walking the object graph. Took about 4 hours total (including the site).
Similar to your camera, we would never have been arsed with doing that much work were it not for LLM vibbing. It allowed us to get something working (sufficiently) very quickly.
At what point of maintaining this code would it be cheaper for me to actually find underlying patterns and program this myself with some actual thought, rather than continuously "shaping the clay" of my LLM slop? These questions keep me awake at night (not really), and probably predict the future of software engineering!
I'm curious, was there a specific reason you went with that solution for grabbing the games assets?
Because asset-ripping is a pretty widespread, well documented practice in gaming - it's how a lot of modding communities figure out and expand on the games they mod.
A quick search shows WH40K: Rogue Trader is built on Unity too, so any number of existing Unity-specific asset-ripping tools should have worked for your purposes. Though I should say I've never actually dabbled much with modding Rogue Trader, so maybe I'm wrong?
It was January I think and I don't remember our thought process. I think we mostly just remember looking at the fandom.com or whatever crap was there and thinking how out of date that stuff was.
We also just wanted to try some modestly sized project with an LLM. Neither of us had any real experience doing any of the things we ended up doing for that project (other than my making the website and him understanding the graphics pipeline).
So, unfortunately, no, there was no specific reason I suspect.
The churn is rewriting 50+ line comments on every function. These comments will also include every place that references this function, as well as random historical data.
Simple, they want Recursive Self Improvement and worry that git will be the bottleneck. So they are making source control as efficient as possible in preparation for that day.
From the first paragraph: "Claude is able to autonomously iterate on research loops, micro-optimize complex programs, and build new services from the ground up at a speed and scale we haven’t seen before. The bottleneck, now, is all of the developer infrastructure built for human scale: source control, builds, and CI."
I find the accelerationist mindset utterly ridiculous. All that busywork and for what exactly? What effect has all this overproduction of LOC had on the physical world and on our existence (beyond the externality of erecting huge data centers in every corner of the globe and the consequent ecological impact)? Is software getting better (in terms of quality, security, efficiency)? Are we getting better government services? Are we getting better food? A better environment (silly question)? Are we working less, as tech titans keep promising us? Are we getting anything better? Not to mention all the negative aspects of this technology and the financial apparatus driving it.
I agree with the sentiment, and the answer to all these questions is usually ‘just wait 6 more months’. Like what is tangibly, actually better in the real world right now with AI? Whenever I ask this question people pipe up with a side project or a new SaaS, but that stuff is just not very important compared to the things you called out. Food, services, QOL.
It seems to me like the AI labs are just pushing the envelope with blind faith that”something” will happen if they just get enough development speed. Nobody seems to have a coherent idea of what that something is.
> Like what is tangibly, actually better in the real world right now with AI?
I took a photo of a tree with chatgpt recently and asked it what species it was. That was kinda neat I guess. Someone forked a ps4 emulator with a bunch of Bloodborne specific improvements so its more playable on my setup. It's kind of okay for medical stuff.
Seems to me the goal is to create a convoluted spaghetti mess in every code base so that humans no longer have a chance to understand how it works and thereby force all software development to go through LLM obfuscator-deobfuscator cycles.
All that busy work was for going from zero revenue to $50B in revenue in about 3 years, and releasing AI systems that are getting better and better at a ridiculous rate. AI isn't solving all the world's problems but you can't deny that they are extremely successful with this method of operating.
I think their success is pretty debatable. This is a company that lost 8 billion dollars last year, is in an extremely expensive race with other companies, and is basically the premium version of something that's now becoming a commodity. They might be doing some impressive work but as a company their outlook seems questionable.
Since when have we expected companies scaling that fast to make a profit? The fact that they've had profitable quarters already is pretty insane.
There are definitely risks to their business long term from commoditization and a lack of a moat, but I don't think that detracts from my original point that they are building something highly desirable with this new version control system today. They are probably the fastest growing company of all time in terms of revenue and valuation? They're not just slopping out more and more LOC for no reason, which was the original point being discussed in this thread.
It's a fair question to ask. If literally all the economic growth (and then some?) is going to AI companies, then nothing else is improving, in the aggregate.
The broader question is definitely relevant, I was thinking more in context of the original post justifying their need to re-invent source control.
I definitely think that's an interesting question though. I know that in the physical product development world I'm seeing really odd asymmetric speed ups. Creating simulations is essentially automated at this point, stuff that would take days to code by hand can be done in an hour. Firmware debugging feels 3x faster than before. But the big slow stuff like manufacturing, assembly, and testing are completely unchanged. All of that little stuff probably adds up to maybe a 5% faster product development cycle than before if you're really leaning into AI. It's the type of thing that will compound over time especially if it's happening all over the economy, but won't blow your socks off.
I think part of the concern is around “could it be better”? Like if an elite athlete eats junk food all the time. Sure it’s working, but is it because of the candy?
Anthropic is often criticized for their poor UI/UX. The models are undeniably powerful but the packaging is typically poor.
This is not to drift into the conversation of profitability either, to be clear.
But the revenue is just because they’re selling at a huge loss and are being allowed to do so because lots of stupid investors are betting that one day they won’t be selling at a loss. That’s not economic growth. Anyone can have revenue selling diamonds for a penny
To play the devils advocate here (just for software), I would say principally this:
"A world where starting is incredibly easy and finishing is more complicated than ever"
LLM's do allow you to oneshot huge starting bits of work in record short time. The frontier of the amount of work they can oneshot is clearly getting larger and larger. But I am still not convinced that they allow you to maintain existing work at a cost that is below a human being.
But, if the frontier of what you can oneshot continues to grow unbounded, then it really will not matter. We will eventually oneshot all software and maintenance will be a thing of the past.
The funny part is, no one is making any money off this either. Traditionally we’d say that profitability is not necessarily an indicator of real value for humanity, but with LLMs we don’t even have profitability! The only people making more money here are manufacturers like Nvidia and data centre companies
> I find the accelerationist mindset utterly ridiculous. All that busywork and for what exactly? What effect has all this overproduction of LOC
Yeah I wish this whole energy was towards optimization and improving hardware.
One thing that people are finding hard to believe is that the frontier labs are cultists who want to "create god". Hence the focus on improving intelligence so they can get to recursive AI, as well as the fear-mongering.
Aanthropic-openAI duo cannot go bankrupt soon enough. This industry is going to flourish with those two out of the picture. They're the pets.com of 1999 trying to have a say how the internet should be.
What defines a "real problem" but that someone is willing to pay money for it?
Who decides what the "real problems" are?
Imo most of the economy and wealth creation is around activity that is not solving a "real problem", but who am I to say if someone else is willing to pay for it? Presumably, for said person, the problem is "real" enough.
Yikes. So much evil done in the name of God. I would think nothing would be further from the conduct of a "moral actor" than equating themselves with God.
Completely unnecessary. Secular humanism for me, thanks.
> ..., the latent demand for merging code is practically infinite, and every bottleneck we remove immediately leads to a further increase in the amount of code we produce.
If the goal is to write software which solves real problems, how can the demand for code grow infinitely?