+1, even the strongest models still produce things that (I suspect) if it accumulates over time will cause issues.
Is my suspicion wrong? Are models getting better fast enough that they'll be able to add new feature (or fix bugs) and clean up even despite a mountain of AI tech debt [1]
[1]: I say AI tech debt because it tends to have some characteristics that are different from human tech debt, such as apparent lack of intentionality (which makes Chesterton's fence evaluation much harder). Perhaps if we make the AI write down somewhere, as briefly as possible, why it wrote each chunk of code...
> Perhaps if we make the AI write down somewhere, as briefly as possible, why it wrote each chunk of code...
Well-configured agents do, in comments and commit messages and docs.
But many people are just stuck thinking in GPT-3 era, where the prompt was all there was to it. It's not. You write a 1000 token prompt describing what you want, and the agent pulls in 10k tokens of system prompt, then spends time exploring the codebase and calling tools and attending to 100k of more tokens from this, possibly including output of sub-sessions that do the above, and somewhere along the way, having read lots of code, run lots of tests, perhaps doing a spike or two, the agent then decides some chunk of code needs to be written in this way, and not in some other way.
The reason is not contained in your original prompt at all. It's almost entirely based on what the agent learned about the current state of your codebase and environment.
And then it reads all the AI-written "docs" from the previous changes, at which point it's all so muddy, it just tries to copy the style and output more junk and more verbose "docs". This is why I never let AIs maintain documents on their own.
And agents that fail to comprehend the context correctly get stopped quickly.
--
But let me spell out my original comment, to clarify:
Agents are like interns: they come to do the job, having none of the knowledge specific to your codebase and company. They have no time, to properly study the whole thing. They get a task, and are expected to figure it out (with help of documentation and people). Then they do the thing, and regardless of the outcome, are gone few months later, and you never hear from them again.
(In many companies, for various reasons, juniors are effectively interns with larger context window.)
In large companies with large projects, everything is moving so fast that no one - no one - can possibly understand the whole thing, and the work paradigm is purposefully structured so they don't need to. In some sense, they're like perpetual interns - each time they start working on different area of the project, they know next to nothing about it, and do not have a chance to fully comprehend it. They just need to figure out enough to deliver on their tasks.
(I know, because I like understanding things, and in one place I was told explicitly it's a common trap for people like me, and I need to learn to suppress it if I am to be effective at my job.)
Unsurprisingly, this works. Both examples are proof positive that this way of working is sustainable and scales well. It may not be optimal, but it works, which means agentic version of it has every right to work and scale as well.
Literally nothing in this wall of text has any televance to what I wrote. And it has a bunch of baffling statements:
> Agents are like interns: they come to do the job,
They don't come to do anything. Stop this infantile antromporphising of LLMs.
They are nothing like interns. Interns learn. And companies don't develop products by getting a new batch of interns every day.
> Unsurprisingly, this works. Both examples are proof positive that this way of working is sustainable and scales well.
No, it's not even close to proof. The conclusion literally doesn't follow from anything in your prevuous text.
If your agents are "perpetual interns" then this is proof that no, it doesn't work and it doesn't scale.
As for large companies... You don't have to know the whole thing, but you do have to know your area. You don't come into work at a large company always asking "what is it I am doing" from scratch every single day.
Replace "day" with "three months" and it should click.
> They are nothing like interns. Interns learn.
And then they go elsewhere, so no, from POV of the company, if you treat "internship period" as "session", they don't learn. But they leave behind documentation and changes to code.
I guess another way to put my point is: the idea of code being self-documenting isn't completely flawed. It works to a degree; in fast-moving teams in large companies, you don't have any other ground truth anyway (documentation is always stale, design docs become obsolete the day they're approved, etc.). People manage. And so can agents.
> but you do have to know your area
It so happens that LLMs know nearly every single area of human endeavor better than most practitioners :).
> Replace "day" with "three months" and it should click.
No, it shouldn't. If you compare agents to interns then you have a new batch of interns not even every day, but every few hours (with every new session, and every new subagent).
> so no, from POV of the company, if you treat "internship period" as "session", they don't learn.
So,
- interns are a very tiny fraction of a workforce at any given company
- no company is ever building the bulk of their products with interns alone
- from POV of the company interns never learn, and the new interns should be taught everything from scratch, again
Somehow, from all this, your conclusion is that "it's proof that it works and can scale"
> It so happens that LLMs know nearly every single area of human endeavor
They have very approximate knowledge of those areas + this knowledge is rather randomized in the output. On top of that they are trained on publicly available code which is shit. Plus they know nothing (and can never learn anything) form your particular code, and your particular project.
This. You’re no longer prompt engineering you’re now building the factory. There’s lots of tools to do this - rules, skills, hooks, subagents, evals, hill climbing etc. highly encourage everyone to think about building the factory that builds the product, not building the product with ai assistance. And it’s fun again.
tl;dr - With AI code is cheap, but poor architecture and skipping best practices is very very expensive. Markdown files will not solve this problem, but it can be solved.
In my case, “Factory” is hundreds of architectural tests built over thousands of application specific tests, and a couple MD files here or there. The MDs are mostly reference material and common patterns. The tests are what actually enforce feature/module isolation and proper architecture.
Lately, I’ve been getting overzealous with protobuffers and service discovery as well. I would never use these tools so frequently if I was designing for humans. With AI I’ll slap annoying boundaries in to enforce separation of concerns without thinking twice.
Making a new project is a pain at this stage, but you can just point to CI and the general structure in an existing project and AI will clone the important bits. I suspect high level architectural rules with programmatic enforcement will eventually be encoded into tools for easily starting new projects. Kind of like pnpm installing specific development practices and architectures.
I used to do a lot of work with reflection, compile hooks, and dynamic source generation before AI. You can do a lot to compensate for the architectural blind spots of AI if you get creative.
> You can do a lot to compensate for the architectural blind spots of AI if you get creative.
How about code blind spots like endless code duplication? Reading the whole DB to read one record? Slow algorithms? Bad data access patterns? Excessive defensive programming? Inventing new issues just because your "high level architectural creativity" mentioned something off-hand? Etc. etc.
On top of that, where are your "hundreds of architectural tests built over thousands of application specific tests" coming from? Also written by LLMs I presume?
> How about code blind spots like endless code duplication? Reading the whole DB to read one record? Slow algorithms? Bad data access patterns? Excessive defensive programming? Inventing new issues just because your "high level architectural creativity" mentioned something off-hand? Etc. etc.
All of these are endemic problems in this industry, openly discussed for decades before LLMs were a thing.
But the pragmatic reality is the same for people and agents - those "mistakes" are common; juniors make them because they don't know better, seniors make them for good reason (pragmatic tradeoffs). Good teams know how to prioritize, and generally fix them when they start causing problems. Agents are very much capable of the same, fully autonomously these days.
Personally, I've had plenty of cases where an LLM came out with a sleak, and arguably over-optimized algorithm or DB query - but when I looked closer at what it did, I saw it first came out with something dumb and claimed a win, then run it, realized it's stupidly inefficient, and optimized it on its own, before giving the final version to me. All in a single turn, without being explicitly prompted to work like this.
Nope you are 100% correct. All of the models will code themselves into local maxima then spend more and more time chasing their tails as every change introduces new bugs, the agent adds more complexity, and the code base just spirals. Best case you end up with a bunch of parallel implementations that all work slightly differently.
It helps a lot of you pay attention to what it is doing and impose order, then regularly spend time refactoring - even if you are having claude do the work itself - you have to spend the time on the overall architecture and design.
I just experienced this with an internal productivity tool. Without claude this tool wouldn't exist - there would never be the justification to take the required time to build it. On the other hand all the interlocking refreshing rules for sourcing data from its inputs got so byzantine it was impossible to add new features or change anything without causing new regressions. Claude built a web of ad-hoc timers, refresh kick-offs, monitors, etc to solve each feature problem or fix some bug but the whole thing was impossible even for claude to understand. It took a special large workflow session to find most of the mess and even then during the refactor it kept finding new things.
I had to stop all feature work for going on four weeks and burn a ton of tokens to build a refreshing/syncing engine that could track dependencies, priorities, etc and fully test it and just now today I am finally able to get feature work going again.
On another feature an agent came up with a state object containing 55 inter-related fields with all kinds of interlocking dependencies. Something so complex no one could hope to understand it. And I find the agents aren't really much better at understanding such things than humans are. If you leave it like that the agent will just perform worse and worse on that part of the code as time goes on. It will make assumptions that force it into bad design corners, miss edge cases, and flat out miss things no matter how well you prompt. Agents benefit from encapsulation and simplicity just as much as humans.
- memo doesn't need to be built (its a shell script)
- once keeps output cache in a running daemon. By contrast, memo stores content under /tmp with whatever the best compression available is (it prefers zstd). There's trade-offs in that. More pareto-optimal from the security front may be to have a sort of session key and to compress-then-encrypt files on disk.
Hey! I use your shell script after seeing it on HN awhile ago. It's very handy (I find myself reaching for it as if it's a built-in) and I appreciate its design (I learned a few things reading it)!
> All those things are true of Java/Kotlin, except even moreso.
The parent did mention:
> ...and is simple to deploy.
Also, my experience with Java programs is that the memory overhead is even higher than Go (where it's a ~2x of what the program holds as heap due to the GOGC=100 default).
It's not harder to deploy in my experience. When people say this they tend to imagine deployment as "scp binary user@host". Well,
./gradlew installDist # build the app for deployment
rsync -avz --delete build/install/my-app/ user@host:my-app/
Just repeat to upload new versions. rsync vs scp isn't harder and the Java version will be faster (incremental). If the host doesn't have a JVM installed, ok... apt-get install one and your distro will keep it up to date. One command.
But in reality most software isn't deployed by copying binaries around. You'd want it to be at minimum run by systemd or kubernetes, for example. And if you want a Docker container then it's pretty easy. Your framework probably configures it out of the box:
./gradlew dockerBuild
Push to the host and start it up.
The reason it's not harder in the end is that the above takes cares of many annoying details that crop up in real deployment, like knowing what CPU and CPU extensions does the host have? Can you deploy incrementally without recopying the whole thing or does that not matter?
The above is for servers, but it's not really harder for CLI tools either. Fat JARs exist. If the user doesn't have a JVM, once again, they can install one easily from their package manager and then it's done - no need to create and distribute half a dozen binaries for all the different OS and CPU combinations that are out there.
But if you want to make AOT compiled binaries and get lower memory usage too, there is GraalVM which can do both. You've got the choice.
Note how the above debate isn't changed by AI in any way. Their weaknesses remain weaknesses, their strengths remain strengths. I wouldn't personally use Go because of its poor feature set and debugging support (errors don't reliably create stack traces). But the arrival of LLMs changes nothing about these preferences and choices. At most you can talk about token efficiency, in theory, but the attempts to measure the real world impact of that don't seem to have yielded decisive evidence.
> And compared to Go, the production observability is very good.
Can you give some examples where Go is lacking and Java is great? (Ideally builtin things, and if not builtin, then things where Go doesn't have an externally developed alternative).
> I also use Clojure and can observe my running app with a full REPL.
Does this work with other JVM languages, like Java, too?
Clojure can to some extent help you get a full/true REPL into Java and any JVM language, but only to some extent. You'll have to use Clojure to get the full observability and program runtime-mutability.
> The OS should also allow marking pages as piority to stop them from being swapped out.
This is IIUC possible using the mlock(2) family of syscalls: https://man7.org/linux/man-pages/man2/mlock.2.html. (On Linux, though I'm guessing other UNIXen and operating systems have it or something equivalent.)
I ordered a Xyber NAS (N150) back when they were still in stock, wiped the Ubuntu that came with it and built up a NixOS config [1]. The power usage is very good (12W with 2 internal SSDs, bluetooth, wifi and a bunch of services running).
If it dies, I won't be too sad since it was about $210. But it's doing pretty well (24/7) so far.
[1]: This ended up being useful. The built-in eMMC chip that I had installed the OS on gave out after ~1 month of usage and I had to reinstall. Thanks to the NixOS config, it took me much less time.
> For example, one of the unique features of my harness is that the filesystem is completely virtualized and all filesystem I/O is part of the session...
What are the mechanics of this? You mentioned bash so I asssume you don't "control" all the tooling. Is it running in a VM with an overlay filesystem that takes snapshots at every file interaction? Or perhaps outside a VM but containerized with a FUSE-based overlay filesystem?
It's not a VM; it's containerized with a FUSE-based custom overlay, and the agent never touches a real mount (with two exceptions: passthrough read-only mounts for things like /bin, and ephemeral tmpfs mount at /tmp).
The main risk is not that the agent breaks out of their sandbox, although it's an infitesimal possibility (that increases to certainty if you increase the token budget to 100M$ and use subagents to spend it all at once).
The main risk (for you, mere human without a 100M$ budget) is that it downloads a compromised package or vulnerability, or that it runs a stray rm -rf or some stupid stuff. A user provides reasonable protection, you don't NEED a vm (and even then that can be escaped).
One thing I'm wondering about (and was happy to see) is how you seem to have your (childhood) friends close by, at least on occasion.
I live in Europe and am considering what a move somewhere else in Europe (+/- 1000km) would mean. Seeing much less of my long-time friends, I imagine. But Austin seems much further from the earlier places you've stated you've lived. Did some of your friends just move with you? Are they visiting often?
reply