Around me everything is at least hybrid, with nobody complaining if you want full time on-site. More and more companies are moving back to on-site full time.
Yeah that's all I could tell at the time. Things were still snappy enough at my low usage, so I just assumed it was my older hardware being a bear. But it was definitely bumping up my power draw unfortunately.
They mentioned this on the podcast, the problem is that since they're development tools the networking stack is still scaled for a full rack with pretty beefy switches.
Either they need to support a whole second network stack with firmware etc which they don't want to do now, or a racklette would cost way too much and almost nobody would pay the price.
I court see a lot of small businesses wanting one. Or as an addon to their existing Colo setup.
Could also be great for DR sites and smaller remote offices. Power is a big factor in deployments and I can see a smaller little turnkey setup that uses less be very popular
Every time I stop reviewing the code from Claude I later find janky crap. Not every review finds janky crap, but often enough that I can't stop reviewing everything.
This is my disconnect with this kind of speed, how are they avoiding the janky crap? Or are they just accepting it and hoping it won't be too crappy or be found soon enough?
I don't have any insight into Anthropic but it seems like that's the case which is just depressing.
> This is my disconnect with this kind of speed, how are they avoiding the janky crap? Or are they just accepting it and hoping it won't be too crappy or be found soon enough?
Using Claude code for a few days gives the answer pretty quickly: they fully embrace the janky crap and ship it to their users without second thoughts. And as long as their users don't mind using their piece of shit, why should they do otherwise?
How can you measure this? It's a cli where I type text in, and the tool fires it off to the model and runs commands locally. The model might be slow, but how would I notice if the cli was slow? Serious question.
For a while just having an indle conversation open in the background would make my PC go very hot. This seems fixed now but that's definitely not the only performance issue I've faced.
Also, the fact that they managed to leak their own code base, twice, tells you all you need to know about software quality at Anthropic.
The shiny lighting animation that runs on the spinner verb runs at an abysmal fps. 20 fps max, and it stutters quite often for me. Not important functionality wise, but it is quite ugly.
Engineering is the science of quantifying risks and costs. If you're not able to quantify the important aspects of a design before constructing it, it's not engineering.
If by this you mean that engineering is about meeting specifications. If by this you mean half assing some janky shit that barely clanks over, then no.
Engineering is about finding the best solution for a problem given time and money budget constraints.
Not throwing endless resources and energies at a problem "because Twitter told me what the best practices are".
E.g. Maybe Claude Code could use half the memory, maybe, but maybe the correct estimation is that solving the memory problem would negatively effect development speed, or would cost many millions of engineering effort and no real world user would really care whether it's using 50 or 500 MBs of ram.
So no, you're not an engineer but a "developer", if you think that "quality" code is engineering, code has a development and maintenance cost, specific directions come with important trade offs.
But since it isn't (we know real users care and complain, we know it's not your made up "endless resources" and "millions of engineering effort" because "millions" of other software products exist that have been made without those) you're short of an actual argument.
> code has a development and maintenance cost, specific directions come with important trade offs.
Yet again you forgot to make a case how any of those trade off towards the current state.
In general, how would you ever determine whether the current situation is justified or not if nothing in your strawman questions anything, but just comes up with not very plausible justifications of whatever already exists???
I can see why anthropic, who prevents their API subscription from being used by other harnesses, can ship such ramshackle bullshit as Claude code with no review.
That has no bearing on the product i maintain, where users very much have a choice, and we keep them with us by solving their problems and listening to their concerns.
But real users do that about pretty much every piece of software, correct? Free SW, expensive SW, open source, closed, SW by a single person, SW from a massive company, etc.
Incorrect, can't complain about 500 memory use if your app uses 5. Also don't get the relevance of the type of developer here in a conversation about specific deficiencies of software itself
I don't think people having the inner desire for excellence would work in the kind of environment that is Anthropic, let alone ship AI slop (as opposed to nice code, AI generated or not).
The process is actually being broken down into enough small steps that there's less misalignment and drift.
There are also some pretty detailed videos out there on YouTube of Anthropic employees showing how they are using cloud code on several orders of magnitude better than where the bulk of usage is.
it seems rude so i dont bring this up when a coworkers tell me their vibe coded prototype is working and theyll never look at the code, but how does someone ignore that they charge per token?
as long as something works "good enough" and the code will never be reviewed theres only incentive to blast through tokens generating superfluous code. god forbid they actually ship a vibe coded product then they can basically ransom you for tokens
im probably being overly cynical but it sort of just seems like standard capitalism to me
Most of the world isn't perfect, and trying to be perfect means you get nowhere. In most places, excessive perfection is implicitly avoided through resource constraints.
In the end, what matters is whether something fulfills its purpose, not how it does it. Code quality, understanding code, etc. are just a means to this goal.
Some level of poor quality is acceptable, and I assume they are hoping that model development (i.e. its capability to fix and clean up an increasingly convoluted mess) will outpace the tangledness of the code.
So far, it seems like they are winning. Yes, the UI is often broken (e.g. for a long time switching away from an ongoing conversation lost it), but it wasn't broken enough for people to switch to other companies that may have had a better UI but in exchange for that a less developed product. If Anthropic instead spent its resources on making the UI not janky and code reviewing everything, the overall product would have likely been worse because they couldn't build and improve features as quickly.
I've seen the flip side where "code quality" becomes its own independent goal and more time is spent gratifying various linters than writing the code, and the inability to get anything done in such an environment is a lot more depressing than janky UX that can probably be fixed by prompting the next generation model to clean it up.
This is a fallacy. Computers are very deterministic, a well designed program will not have any bugs or unforeseen failures at all. You're not planting a tree and going "oh god I hope it survives the winter", you're programming a human-made computer, made of hardware that undergoes such incredible scrutiny before it's ever powered on that you can trust that it executes the same code, the same way, every single time, for decades.
No idea why people who clearly don't understand the difference between poppy farms and computers come in here and claim the world isn't perfect and so we should all accept sloppy shit code that barely works. The world isn't perfect, but software CAN be quite close.
Every single instance of jank on computers is a bug that someone wrote, or an LLM wrote, that can be found, understood, and fixed. And only once you understand this, can you talk about software quality to other engineers. You need to understand that the baseline is that software is not fallible the way the rest of the world is.
See sqlite3 if you want to know what nearly bug-free software looks like. I can't point you to a potato farm or an artist who is perfect, but I can point you to that and say "this is where we go when humans do a good job".
> a well designed program will not have any bugs or unforeseen failures at al;
You can only say that of either a very simple program or perhaps one with formal verification (which is overkill for 99.99% of software that gets written).
Look at the Linux kernel, arguably a well designed system (unless you ask Tanenbaum) with top notch developers, and still is full of bugs and security holes.
I agree with your general sentiment on bugs. Id personaly go one step further and say:
Every system in the world has bugs and security holes. If its exists, its buggy because "bug" is very loosely defined as "doesnt exactly work as expected". The security topic is even more complicated.
But I also would like to point out that comparing the Linux Kernel quality and development to Claude Code is surely a joke. The former is an ocean of features, compatibility and history. The latter, in the end still one executable CLI tool calling endpoints.
I have never in my career seen a (useful) program without bugs. I'd go so far as to say that, at least for most software, the optimal amount of bugs is not zero - if you write your code carefully enough to never have bugs, you will ship far too slowly. The point of code is not to be beautiful and perfect and well designed - it is to solve problems in the real world.
By the way, all hardware has bugs too; we just don't see them often because the programmers building the stacks that we run our code on work around them.
I want to agree with you, but most software projects people work on are ridden with unknown bugs and people do in fact live with this. I'm not aware of any commercial software project that arrives at the goal "not having any bugs or unforeseen failures at all". And where this does happen (space probes?) it is very expensive.
IMO the issue with LLMs is they massively widen the "unknown unknowns" (skipping review makes this problem even worse). This is obviously much worse when an LLM is actually a part of the software rather than just producing software that is otherwise deterministic.
I think OP was a little too absolute but i agree with the premise. People who engineer buildings and bridges are not allowed (generally) to ship mistakes. Software "engineers," though, have just accepted it as standard practice.
I do think the world would be better if commercial software had to meet some professional standard of fitness, with meaningful repercussions if not. I also realize that ship apparently sailed before i was born, and I'm not paid to do serious engineering, I'm paid to ship features.
(Not even saying I'm qualified to do the former anyway, i just also think every time national air traffic is shut down or like in France the other day and hundreds of supermarkets lose all their frozen inventory, or i can't do my job because some service failed, or yes, claude code is slow and heavy age can't work right in every terminal, that maybe we shouldn't be writing the stuff that the whole world relies on to function every day with zero agreed upon professional standards)
> I'm not aware of any commercial software project that arrives at the goal "not having any bugs or unforeseen failures at all".
It's ludicrous that we have executives at tech companies that think this is possible.
One S&P 500 company I worked for signed a big contract to deliver bugless software for an enterprise web portal for the state of Minnesota. I was on the team that had to make that magic happen. I had to leave the company not long after that due to nearly life-ending burnout.
So I would throw scorn at the ivory tower inhabitants here who should know better than to suggest that anything of substantial complexity can be bug free
Exactly! That's my point. It's already a problem, and it's not "fine" just because it's the status quo, but at least we can chalk it up to "humans are fallible". LLMs 100x the amount of mistakes a programmer can make in a day, and that's inexcusable to not take seriously, that's my point. We don't need to chase perfection, but we can't pretend like whatever Claude one-shots is any good at all.
> LLMs 100x the amount of mistakes a programmer can make in a day
1. Have you seen that? If so, which agent+model?
2. Is “per day” intentionally used instead of per implementation - i.e a coding agent may A) be far faster tok/s than a human, and B) work more hours per day, so it’s similar bug rate per implementation? Or did you mean bugs per implementation?
1. Are you referring to LLMs handling 100x as many patches as a human can, or that LLM implementations are 100x as long as a human's implementation?
2. In my experience, LLM implementations are often 2-4x as long, largely because they have a lot more error handling than I would have chosen to implement. However, I don't find they have correspondingly X times as many mistakes.
3. Part of how agents seem to avoid many mistakes or even catch mistakes is by testing things. For example, it caught issues in Pester functions on certain Pester versions, or will even analyze binaries to verify the right config settings to use, etc.
4. If you are seeing 100x as many bugs, which agent+model?
LLM is clearly not 100xing the mistakes as we have witnessed Anthropic’s success. Otherwise the system would not have been functional. It’s not like their system is not tested.
The thread is about jankiness, lack of feature cohesion and code quality.
What we are discussing here is: is it acceptable to accept poor code quality and trade it off with delivery?
> What we are discussing here is: is it acceptable to accept poor code quality and trade it off with delivery?
I found that people that lean towards the latter also favor profits over customer satisfaction. Then they will try to defend themselves with a black and white view “it’s either this scrap or no delivery”.
> This is a fallacy. Computers are very deterministic, a well designed program will not have any bugs or unforeseen failures at all.
I don't know where you think you've found a fallacy. Computing systems are expected to interact in various ways with the outside world, which does include poppy farms, and people, and other decidedly imperfect things.
Yes! But they can behave deterministically for any kind of input data. Of course they interact with the outside world, and I'm not saying you can't have errors, I'm saying they're not random and driven by god's will, they're programmed explicitly.
In my first class of Software Engineering, I was taught that software is designed for changes. Even if software the fully deterministic, it interfaces with a world that is constantly changing: requirements change, customers change, political systems change.
Unless the chip has a bug or manufacturing error; I've worked with several fleets where small numbers of chips had stochastic errors. I doubt there is any fleet which does not periodically have some level of corruption, which typically goes unnoticed unless you have engineers paying close attention and some level of verification.
> Reporting bugs, patching systems, updating software is common to most software.
Yes, but this is a side-effect of trying to do good work and writing code that is understood and works.
When you ship sloppy, AI generated code, that no humans at the company have read and understood, you're missing the main part about engineering and are skipping straight to the life-support stage of the software, and you treat the (previously) side effect as your main job. Reporting bugs and shipping patches becomes your job, instead of engineering software, which happens to also come with that task.
I expect people who are skillful and enjoy writing good code, are among those who dislike code generated with AI tools, and perhaps feel they have had a role or pleasure denied to them.
Those who write shit code, now feel that can produce vastly better software in a fraction of the time, and feel a sense of accomplishment.
Not sure how that maps to software engineering at all. Seems more like a straw man, did you read my comment at all or just the first words? My point is that the mostly deterministic machine is very much not "imperfect" and doesn't need to be cuddled and watered to work. You simply write code, and if you're like a LITTLE BIT careful, it works super well. It's really not rocket science.
Nobody has to make the perfect program, but we can't chalk it up to "the world isn't perfect" when I click the windows start menu and it opens empty (an existing bug btw)
This is a category error. "memory" is not a computer, so the existence of ECC proves nothing about computers. Concrete is not deterministic in the real world either, and we're still able to control the shape of structures and make them last for centuries.
Code quality - and its best friend, interface design - pay dividends in large codebases. The fastest moving and most effective organizations I've worked at are ones that focused on code quality from the beginning.
All software is broken, but LLMs seem to make the wrong compromises consistently when working in large codebases.
> The fastest moving and most effective organizations I've worked at are ones that focused on code quality from the beginning.
I agree with that in the pre-LLM world, but I don’t believe it’s the same today.
You can build something with “bad” code that works and is full of technical debt, but today paying this debt is getting cheaper and cheaper.
So if Claude’s wrong design decision leads to having to refactor several chunks of a codebase in order to implement a new feature, does it really matter if it’s Claude who is going to refactor it in a matter of seconds? Especially for a company like Anthropic which has a virtually unlimited number of tokens to spend on it.
Sounds like converting technical debt to monetary debt that will still ultimately need to be paid. Pay engineers and/or anthropic to fix the problems caused by anthropic. What do you do when anthropic servers are down?
Now instead of the AI agents swarm you let 100 engineers making the changes on that large codebase. How do you keep the code "quality", which is a very subjective thing, while at the same time, as you say, keep the "fastest moving and most effective organizations" trait? Impossible without gatekeeping and thorough review process both of which contradicts your trait.
Good interface design allows extending or changing component X without having to step on the toes of your colleague working on component Y downstream. It's absolutely essential.
Interface design is one of the things Claude is worst at in my experience (using it every day for the past year).
That reads as an example from GoF era. In practice, especially with complex software engineering codebases, which I thought you were implying by saying "large", these patterns barely exist because the problems being solved are often in friction with what is considered to be the "best practices" elsewhere (in the books, other much simpler codebases, etc.).
Thank you for engaging with my point! I agree, actually. I think if good design patterns could simply be copied out of a book, LLMs might be better at software engineering.
My point is more about up-front investment paying dividends. To speculate, it seems like the reward hacking is not there for long-term thinking.
But a little bit of directing of the architecture can easily bring in design patterns.
Ideally, you design patterns that enable quick replacing of parts that get to messy. The code is cheap, just start from scratch with the defined functionality pre defined instead of bolted on with jank.
It's true though. I've also experienced the difference between a codebase that was an absolute tarpit from the start, and one that obviously focused on quality from the start. They both became jankier over time, but one notably less so. The latter was a bigger, more successful organization. Does that have anything to do with their codebase being cleaner? Maybe. Maybe not. It was certainly easier for me to work in.
I would argue that the "fastest moving and most effective organizations" trait is relative to what we find in the set of software develepment groups. There must be a difference in code quality and development speed between the best and the slowest player, and there must be some parametrizable difference in standards (if there are any) the players operate with.
This is too vague for me to answer but I'll try - effective organization is the one whose revenue and net-profit per employee is higher than their competition. And fastest moving organization would be relative and tightly coupled to being an effective organization too. Reminiscent of a fail-fast culture without red tapes. Software "quality" is IME rarely a discriminator.
> Most of the world isn't perfect, and trying to be perfect means you get nowhere.
That's a thought-terminating cliche. When someone asks how can they can avoid generating a pile of slop (which will compound over time and get increasingly harder to deal with), they're not seeking "perfection", and framing it in those terms is dishonest.
Furthermore, can we step back for a second and remember that user and developer experience matters too?
Working with code like that sucks. Spending much of your day on polishing a turd that grows more unwieldy with every iteration is going to get old, fast. And I also think that's going to take a significant mental health toll on people eventually.
The business won't care about this but we should. AI was supposed to make things better, better quality, better reviews, better QA, better product, better UX - remember?
> janky UX that can probably be fixed by prompting the next generation model to clean it up
I'll counter by pointing out that "Next one will fix everything" has been a meme for at least 2 years now and the slop just keeps getting sloppier.
They're just accepting it. It shows in the quality of their products also. They ship a lot of features in Claude Code for example but it is clear that is a vibe-coded mess.
Mess might not be the right word, but lots of features are clearly developed without any grand vision or coherent thought about how the features are supposed to work together.
Did you hear the stats on the claude code leak? there were multi-thousand line for loops with hundreds of conditional / if statements written in typescript. Its insanely difficult to decipher. At this point "mess that only AI can understand" is what they have and must continue building upon.
Also, we don’t know how many billions of dollars was spent on making it work. They are running 1000s of agents against their codebase 24/7, and still only producing what they’ve put out so far.
It might be fully automated and eventually the agents figure out all of the bugs, but I suspect the costs are so large that no sane company would ever write code this way, and probably WAAAY more expensive than just hiring humans to write and test it.
Huh? Yeah, of course they can do what they want. That does not preclude me from having an opinion about what they should prioritize. I am criticizing their product decisions, not claiming they are not free to make them...
Haha, that's a good one. Perhaps we should coin a new phrase: "dog shitting", where a company eats its own dog shit before inflicting it on their users. :)
> By virtue of Claude’s ability to write immense amounts of code much faster than we could ourselves, Scion is among the most well-tested and well-defended codebases we have ever worked on.
> how are they avoiding the janky crap? Or are they just accepting
They're obviously just accepting, have you seen any of their results? No internal insights required for these observations. Check out their clauded home page - try to open the right hamburger menu and click on 'policy' - it's impossible to do from the top because "Research" will expand on hover and push 'policy' down, and when you move further down, 'research' will collapse and 'policy' will jump up.
Likewise, the AI apps are infamous for their poor performance and buggy nature. Correctness is the price of such velocity
And it's not that human-rolled code never had issues like this before, but knowing it's AI being shoveled down your throat is what adds the insult to the injury.
on mobile it works (where I would expect the hamburger menu), and on usual desktop I have the usual menu bar, but when I change the window size on desktop I get the hamburger menu (and then the UX jankiness is evident, but for me easy to work around, as moving the mouse down closes the "research" block and then it's quite simple to move it up to be able to click "policy")
at this point I think Opus does something better by default
why wouldn't it work on mobile? As you hover your finger over the screen, the finger's shadow doesn't generate any events for the buggy website to react to?
By the way, the hamburger shouldn't even be there as there is enough space to fit the full menu (and even the company name) at the viewport sizes I've checked
as I was reading your comment I was thinking about what the bug is, and my initial guess (pulled out of thin ass-air) was that even on mobile somehow the touch targets overlap and there's no way to actually send the click to "policy"
I think at work I am just barely in the category of I've removed more code than I have added. Over many, many bugs and long meetings I've sort of arrived at the conclusion that in most cases, the best code (excluding comments) is the code that doesn't have to exist [0]. It occurred to me at work today while refactoring some change that even in this LLM-driven world, I can still optimize for that for some additional challenge and satisfaction.
The root of this challenge comes from more text aka more code ~= more tokens, so these firms have little incentive to optimize for that mindset. Sure, they might have more optimized individual responses - but the end result is still giving people what they want: more features, more documentation, nice, long, verbose and well-formatted Jira comments, of sometimes questionable but sometimes undeniably impressive quality. The band marches on, and when your codebase outgrows the constraints of the tried-and-true tools, you can just invent your own. With documentation and an MCP or two, or ten -- to also pull into your context window.
[0] I work on an operating system, so a suggested retort is to say this is me pushing responsibility and logic to the people higher up on the stack ;-)
> The root of this challenge comes from more text aka more code ~= more tokens, so these firms have little incentive to optimize for that mindset.
I'm not sure if this is true. The code is such a small part of the total output. Thinking harder, like what would be needed for architecturing better code, would use more token.
+1 my code quality degenerates exponentially after every prompt if I skip reviewing
Current agents never take a step back to refactor and to create a good abstraction as the requirements evolve, they only care for delivering fast even at the price of garbage code which doesn’t scale
An idea to start could be something along the lines of an agent given tasks incrementally, where the agent would be required to extend/generalize a piece of functionality. A possible reward could then be the amount of code the agent had to add/remove to complete the task.
I assume frontier labs will start hiring devs specifically for generating training data for this purpose. It's a skill you can get better at. I don't see in the limit why it couldn't be solved, but I agree, for practical reasons it's somewhat unlikely.
> I assume frontier labs will start hiring devs specifically for generating training data for this purpose. It's a skill you can get better at. I don't see in the limit why it couldn't be solved, but I agree, for practical reasons it's somewhat unlikely.
Because of the scale that would be needed. Like, design stuff is much fluffier than code correctness or functionality, so really really hard to generate enough synthetic data to tune the models to do better.
In the limit yeah it would work, but in the limit we're all dead ;)
At some point on this ride, if devs are really put out of work and wages are massively depressed (along with the devs), then it might become cheap enough to make work.
If you spent some time reading Anthropic's docs on their website, you'll notice that sometimes the same feature has different intended behavior depending on which page you are reading and may or may not match the actual software behavior. You are better off using the docs as inspiration and figure out what the software does at runtime. And I have definitely done the thing where I just ask Claude to reverse engineer its own binary. I doubt anyone at Anthropic can explain what it's supposed to do.
This, so much. Even really important stuff like their GitHub integration, which works completely differently in Claude desktop, claude code from
Desktop client, and Claude code from CLI
+1, even the strongest models still produce things that (I suspect) if it accumulates over time will cause issues.
Is my suspicion wrong? Are models getting better fast enough that they'll be able to add new feature (or fix bugs) and clean up even despite a mountain of AI tech debt [1]
[1]: I say AI tech debt because it tends to have some characteristics that are different from human tech debt, such as apparent lack of intentionality (which makes Chesterton's fence evaluation much harder). Perhaps if we make the AI write down somewhere, as briefly as possible, why it wrote each chunk of code...
> Perhaps if we make the AI write down somewhere, as briefly as possible, why it wrote each chunk of code...
Well-configured agents do, in comments and commit messages and docs.
But many people are just stuck thinking in GPT-3 era, where the prompt was all there was to it. It's not. You write a 1000 token prompt describing what you want, and the agent pulls in 10k tokens of system prompt, then spends time exploring the codebase and calling tools and attending to 100k of more tokens from this, possibly including output of sub-sessions that do the above, and somewhere along the way, having read lots of code, run lots of tests, perhaps doing a spike or two, the agent then decides some chunk of code needs to be written in this way, and not in some other way.
The reason is not contained in your original prompt at all. It's almost entirely based on what the agent learned about the current state of your codebase and environment.
And then it reads all the AI-written "docs" from the previous changes, at which point it's all so muddy, it just tries to copy the style and output more junk and more verbose "docs". This is why I never let AIs maintain documents on their own.
And agents that fail to comprehend the context correctly get stopped quickly.
--
But let me spell out my original comment, to clarify:
Agents are like interns: they come to do the job, having none of the knowledge specific to your codebase and company. They have no time, to properly study the whole thing. They get a task, and are expected to figure it out (with help of documentation and people). Then they do the thing, and regardless of the outcome, are gone few months later, and you never hear from them again.
(In many companies, for various reasons, juniors are effectively interns with larger context window.)
In large companies with large projects, everything is moving so fast that no one - no one - can possibly understand the whole thing, and the work paradigm is purposefully structured so they don't need to. In some sense, they're like perpetual interns - each time they start working on different area of the project, they know next to nothing about it, and do not have a chance to fully comprehend it. They just need to figure out enough to deliver on their tasks.
(I know, because I like understanding things, and in one place I was told explicitly it's a common trap for people like me, and I need to learn to suppress it if I am to be effective at my job.)
Unsurprisingly, this works. Both examples are proof positive that this way of working is sustainable and scales well. It may not be optimal, but it works, which means agentic version of it has every right to work and scale as well.
Literally nothing in this wall of text has any televance to what I wrote. And it has a bunch of baffling statements:
> Agents are like interns: they come to do the job,
They don't come to do anything. Stop this infantile antromporphising of LLMs.
They are nothing like interns. Interns learn. And companies don't develop products by getting a new batch of interns every day.
> Unsurprisingly, this works. Both examples are proof positive that this way of working is sustainable and scales well.
No, it's not even close to proof. The conclusion literally doesn't follow from anything in your prevuous text.
If your agents are "perpetual interns" then this is proof that no, it doesn't work and it doesn't scale.
As for large companies... You don't have to know the whole thing, but you do have to know your area. You don't come into work at a large company always asking "what is it I am doing" from scratch every single day.
Replace "day" with "three months" and it should click.
> They are nothing like interns. Interns learn.
And then they go elsewhere, so no, from POV of the company, if you treat "internship period" as "session", they don't learn. But they leave behind documentation and changes to code.
I guess another way to put my point is: the idea of code being self-documenting isn't completely flawed. It works to a degree; in fast-moving teams in large companies, you don't have any other ground truth anyway (documentation is always stale, design docs become obsolete the day they're approved, etc.). People manage. And so can agents.
> but you do have to know your area
It so happens that LLMs know nearly every single area of human endeavor better than most practitioners :).
> Replace "day" with "three months" and it should click.
No, it shouldn't. If you compare agents to interns then you have a new batch of interns not even every day, but every few hours (with every new session, and every new subagent).
> so no, from POV of the company, if you treat "internship period" as "session", they don't learn.
So,
- interns are a very tiny fraction of a workforce at any given company
- no company is ever building the bulk of their products with interns alone
- from POV of the company interns never learn, and the new interns should be taught everything from scratch, again
Somehow, from all this, your conclusion is that "it's proof that it works and can scale"
> It so happens that LLMs know nearly every single area of human endeavor
They have very approximate knowledge of those areas + this knowledge is rather randomized in the output. On top of that they are trained on publicly available code which is shit. Plus they know nothing (and can never learn anything) form your particular code, and your particular project.
This. You’re no longer prompt engineering you’re now building the factory. There’s lots of tools to do this - rules, skills, hooks, subagents, evals, hill climbing etc. highly encourage everyone to think about building the factory that builds the product, not building the product with ai assistance. And it’s fun again.
tl;dr - With AI code is cheap, but poor architecture and skipping best practices is very very expensive. Markdown files will not solve this problem, but it can be solved.
In my case, “Factory” is hundreds of architectural tests built over thousands of application specific tests, and a couple MD files here or there. The MDs are mostly reference material and common patterns. The tests are what actually enforce feature/module isolation and proper architecture.
Lately, I’ve been getting overzealous with protobuffers and service discovery as well. I would never use these tools so frequently if I was designing for humans. With AI I’ll slap annoying boundaries in to enforce separation of concerns without thinking twice.
Making a new project is a pain at this stage, but you can just point to CI and the general structure in an existing project and AI will clone the important bits. I suspect high level architectural rules with programmatic enforcement will eventually be encoded into tools for easily starting new projects. Kind of like pnpm installing specific development practices and architectures.
I used to do a lot of work with reflection, compile hooks, and dynamic source generation before AI. You can do a lot to compensate for the architectural blind spots of AI if you get creative.
> You can do a lot to compensate for the architectural blind spots of AI if you get creative.
How about code blind spots like endless code duplication? Reading the whole DB to read one record? Slow algorithms? Bad data access patterns? Excessive defensive programming? Inventing new issues just because your "high level architectural creativity" mentioned something off-hand? Etc. etc.
On top of that, where are your "hundreds of architectural tests built over thousands of application specific tests" coming from? Also written by LLMs I presume?
> How about code blind spots like endless code duplication? Reading the whole DB to read one record? Slow algorithms? Bad data access patterns? Excessive defensive programming? Inventing new issues just because your "high level architectural creativity" mentioned something off-hand? Etc. etc.
All of these are endemic problems in this industry, openly discussed for decades before LLMs were a thing.
But the pragmatic reality is the same for people and agents - those "mistakes" are common; juniors make them because they don't know better, seniors make them for good reason (pragmatic tradeoffs). Good teams know how to prioritize, and generally fix them when they start causing problems. Agents are very much capable of the same, fully autonomously these days.
Personally, I've had plenty of cases where an LLM came out with a sleak, and arguably over-optimized algorithm or DB query - but when I looked closer at what it did, I saw it first came out with something dumb and claimed a win, then run it, realized it's stupidly inefficient, and optimized it on its own, before giving the final version to me. All in a single turn, without being explicitly prompted to work like this.
Nope you are 100% correct. All of the models will code themselves into local maxima then spend more and more time chasing their tails as every change introduces new bugs, the agent adds more complexity, and the code base just spirals. Best case you end up with a bunch of parallel implementations that all work slightly differently.
It helps a lot of you pay attention to what it is doing and impose order, then regularly spend time refactoring - even if you are having claude do the work itself - you have to spend the time on the overall architecture and design.
I just experienced this with an internal productivity tool. Without claude this tool wouldn't exist - there would never be the justification to take the required time to build it. On the other hand all the interlocking refreshing rules for sourcing data from its inputs got so byzantine it was impossible to add new features or change anything without causing new regressions. Claude built a web of ad-hoc timers, refresh kick-offs, monitors, etc to solve each feature problem or fix some bug but the whole thing was impossible even for claude to understand. It took a special large workflow session to find most of the mess and even then during the refactor it kept finding new things.
I had to stop all feature work for going on four weeks and burn a ton of tokens to build a refreshing/syncing engine that could track dependencies, priorities, etc and fully test it and just now today I am finally able to get feature work going again.
On another feature an agent came up with a state object containing 55 inter-related fields with all kinds of interlocking dependencies. Something so complex no one could hope to understand it. And I find the agents aren't really much better at understanding such things than humans are. If you leave it like that the agent will just perform worse and worse on that part of the code as time goes on. It will make assumptions that force it into bad design corners, miss edge cases, and flat out miss things no matter how well you prompt. Agents benefit from encapsulation and simplicity just as much as humans.
Because they are using 100x more tokens with the latest best models only.
If you cranked up turns and only used fable level model for every interaction, you would get way way less jank. Plus when you have essentially unlimited tokens, you can always do extra AI review passes to reduce jank.
I think for us, outside of Anthropic, it is hard to imagine the scale of AI agents that they are running internally . We are talking about probably 10s of millions $ worth of spend monthly if not more.
So they are proving that this model of development isn’t really sustainable. I think this blog is a colossal self own “look at what we need to do to make it work!”
I am experiencing the same. I have decided that the only way to use these tools (at least for now) and at the same time to avoid the janky crap and understand what's going on with the codebase, is to ask from the model to work in very small committable batches which I can review without my head exploding, commit in branch, repeat. I feel much better now and I catch things fast and early.
This has become my approach as well. Create a work tree per larger thing / batch of things and manually review everything in small commits, then squash merge the whole tree to keep the history manageable and clean.
Ultimately the "speed" that matters, is the speed of delivering _useful_ improvements over time. The speed of delivering that today, is bottlenecked by the speed users can accept and form opinions on new functionality, and the speed of doing it tomorrow is also bottlenecked by compounded technical debt.
For now at least, I'd say churning code faster and faster cannot widen either of these bottlenecks much.
Today, I use Claude heavily, but review all the code. I'm faster, but not by the multiples people speak of. And that seems ok to me.
> Every time I stop reviewing the code from Claude I later find janky crap
I'm very curious about that because that has not been my experience at all. I used to find janky crap from older models and I had to review every line of code very carefully. With newer models, and especially large ones like Fable, it just doesn't happen anymore. The code that I get is what I would have written. Actually, it's usually better because it covers more edge cases that I wouldn't have thought about.
At this point, I guide Claude Code, provide general direction, and make design decisions, but I rarely write code anymore. The code that I get from the LLM is just better.
I do not fully understand where the disconnect comes from. I do believe people that say that they get crap from LLMs, and I suspect I can guess some of the reasons for that, but I cannot be sure.
> I do not fully understand where the disconnect comes from. I do believe people that say that they get crap from LLMs, and I suspect I can guess some of the reasons for that, but I cannot be sure.
Same here. There is an introspective comment below by khazhoux asking the question from the opposite experience. Why do developers end up on different sides of this coin?
LLM capability is obviously important variable. But those with bad experiences report using top tier models, so that isn't it. Likewise, level of experience can be ruled out. Each side of the coin have experienced developers reporting.
A lot of the other variables are not reported but I can go from my own experience.
Greenfield vs brownfield? The first is indeed usually easier but I get great results updating large code bases with decades of development. And I suspect developing greenfield with LLM is easier as the code is better written from the start.
Rarity of the subject matter? In principle, yet another XYZ app or web page is well represented in the training data so the LLM should have an easy go at making similar. In the past with low-end LLMs, this was noticeable. Working in relatively obscure programming languages produced just garbage. But, that was prior to the inflection point. And today I get good results working on relatively arcane scientific/mathematical algorithm as I do when I ask the LLM to barf out some web app.
Different levels of testing and validation and expectation? Even best LLMs make mistakes. The better LLMs make fewer and expose fewer still to the human as they create internal tests. Maybe people have different expectations and those with bad experience expect perfection from a single prompt? Perhaps, but any experienced developer knows bugs are a part of software development.
What else?
Whatever the answer, it's not a simple one and it may be a confluence of many smaller things. Maybe different developers have a multi-variant function in their brain that pops out a different quality-of-software score and also a different threshold that divides good from bad and all we see in their reports is that binary conclusion.
In my experience, the worst crap gets into the codebase through updates. Especially when dealing with something hard, Claude will take shortcuts, that undermines the project's quality. In my projects, I deal with it by still pair programming with it. But as far as I understand, Anthropic deals with it by re-creating projects, rather than updating them.
They aren’t. Their source leak showed what a mess of a codebase they’re working with. That was, I assume built with unlimited access to frontier models as well.
Code quality is just going to get worse. I think that ship has sailed. Its no longer economical for humans to write good code. We're all going to be building better scanners, automated review mechanisms, quality checkers, feedback routers, etc. for the next 5 years.
> This is my disconnect with this kind of speed, how are they avoiding the janky crap?
Because you are using LLMs to write code that you would be writing by hand.
Use it to write thing you wouldn't have time to write by hand and that are not mission critical: custom linters, support tools and so on. You know, each time you think to yourself "hey, I would write something to automate this chore, if only I could set aside one or two days".
You can also use it to document silly things you wouldn't take time to but that provide context to agents. Ask it to do a performance audit and to give you a few SQL queries and bash commands to gather stats about prod and logs. Then commit this document.
Will you read it? No. But your agent will use it to write code that's fit.
Keep doing this and 1/ you will be more productive, 2/ your agent will progressively improve so that you will let it do more and more.
I've been building some game user interfaces with it, and on the surface it looks amazing. It's iterating at lightning speed, but then after a while you start noticing all the holes.
For example I've made a first screen with a lot of elements, and now I want to create a 2nd similar screen, so I tell it to use the same code as we've created for the first to make the 2nd. With that as input it will create it and
a) use e.g. the variables and setup from the first
b) copy everything into the 2nd
c) copy some things
d) make a generic parent for both to inherit from
A human would have realized that this is something we're going to be doing a lot, so the last is the correct answer. Make generics for all elements, e.g. the screen's panel, effects, buttons, text etc. then implement those along with possible overrides etc. for all.
But it seems that it has no clear methodology. Depending on your exact wording along with today's weather you will get any combination of the options.
They almost certainly don't review the generated code, or at least not with the common techniques we used so far (human review), otherwise their development would be largely bottlenecked by the review process - and I imagine this is where 99% of the companies are at right now.
Does anybody know of "Real" companies that dropped reviews due to immesne use of AI agents, successfully, and their products got visibly better (or at least not hellish worse)?
By "Real" in this context I mean companies that aren't Anthropic .i.e. don't sell any kinda AI product and existed at least 10 years before the AI boom.
Esp. in regulated or at least sensitive fields like finance, banking, medical, logistics etc.
Dropping human-in-the-loop reviews is inevitable - it's just a matter of time. You cannot design a Ferrari and then let the driver crawl its speed by manually pushing it down the road. And all the "real" companies I have worked with are not 100% but 1000% into the AI - the incentive is huge.
While companies are heavily incentivized to speeding up, cutting costs, etc. I expect that friction imposed by the humans is not going to go away that quickly.
I don't think too much of an insight is needed to understand that most of them 30000 code changes per day is just going around in circles, agent A introducing some kind of new abstraction, agent B finding and refactoring that abstraction, agent C figuring out that refactored abstraction introduced regressions, fixing and creating new tests for an abstraction, agent D finding out new abstraction was not needed in the first place and cleaning it up. Management s ecstatic seeing 4 merged PRs – time to make a blog post about merges per day growing more than tenfold.
Have you read the leaked source code of Claude Code? at this point they make sure it works with minimal bugs, don't think it is even tested by a real person, tests too should be vibe coded by agents!
As others have already mentioned, they don’t. Even something as important as payments they seem to care little about, couldn’t give them money for a while due to known issues with their web.
But also, anything coming from Anthropic (and the rest) is 100% marketing.
I think it is mostly this and my experience is that agents feel much more at home here. But on the other hand monorepos disadvantages become even more pronounced with agent work. It will become an unmaintainable mess, even if your agent can iterate through it regularly.
I still think building a pipeline that allows for separation of concerns is a requirement for maintainability. Sure, Claude excels to build software if I just give him a server where he can run wild. Any new requirement is an extension of the old repo.
But at some point you just have a giant mess and I am not really convinced of this approach, even if it certainly fits agents.
People need to actually put in some effort with LLMs here. Ask any SOTA agent to identify a specific type of "janky crap", and it'll quickly find every single occurence. Fixing it with the right pattern should be trivial for an experienced engineer, and therefore its agents.
Just do this for every type of "janky crap" and your codebase converges to perfection.
It's not that hard, but it takes a lot of tokens, and setup.
> This is my disconnect with this kind of speed, how are they avoiding the janky crap?
My understanding is that they have manual reviewers in phase 1 of their projects. Their goal is to refine the automation of reviewing, writing skills/rules, etc, enforcing architectural choices, etc.
Then in phase 2, the review phase is AI-driven itself and humans only get to be decision makers of escalated issues.
Its funny the worst code I used to notice was my old code that solved the problem I had faced at the time but had some flaw in the thought process that I later corrected. The same thing applies to generated code. Yes there is a lot of garbage but you can figure out ways to generate better output as you learn the tools and the models get better
I guess, they have some form of harness engineering optimized for Claude + they likely use more advanced models than those available to us.
And yes, they likely accept that not every line of code is clean.
But honestly (and unfortunately), average source code is nothing stellar either. I do SW dev for 30 years. And most projects I have seen were somewhat ugly.
lot of people here are criticising the set of features stuck together or "janky crap"...I don't know where you work, but I have seen a lot of that in all my career, and still do... and it was done by humans. The difference is that ai is doing it faster and can also quickly change it in the future, so not really sure it's the same level of problems.
This is a big change in how we think about software development, bugs, and features tbh in terms of speed of reaction, identifying problems and caring about janky code or performance, because solving those problems, if they present themselves, is cheaper than before.
Maybe the solution is just to hire someone who likes to fix janky crap full time and have that be their job. Honestly, I'd do it. I hate jank, so I like de-jankifying things. I don't think I'd mind such an arrangement personally.
Many of the comments here seem to stem from the implicit assumption that before claude, software products were thoughtfully designed, secure, reliable, and beautifully engineered works of art.
They get the latest models, they probably don't use the quants they give us, they have unreleased models, etc. We're not in the same situation. Maybe people who use the API idk
“Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code.“
Probably is a source of some bugs. But probably what they’re seeing is that it isn’t a major factor. Rather a bigger factor is agents trying to make changes were the intent isn’t aligned. And note, they don’t say they never look at the code. I suspect that valued human experience is having a feel for what types of changes is AI more likely to make mistakes with.
Many of us have used Claude Code. And Codex too. These tools are essentially produced by bots. What is the quality like? I would say pretty good! Are there occasional issues in the tool itself? Yes. But more than average software produced by humans? No. Not at all. In fact I would say these tools are significantly better than average commercial software.
Pretty good? The list of bug fixes that's shipped for Claude Code every day doesn't fit on my screen. Their issue tracking is a big mess with 100.000 issues in it, most of them auto-closed. People even (vibe)coded websites to track if a new build is usable (e.g., https://claudeissues.com/status) that my Claude now started using to explain its own behaviour.
Your bar for quality must be pretty low if you feel this is better than average?
That's easy fixed by actually taking the effort to go and use other AI harnesses. Youll see that Claude Code somehow works then it doesnt for no obvious reason, then again something else... Like there is no system to it. Its somehow obvious that it isnt developed by humans or at least not as enough as should be.
Have you used any other harnesses? I feel like the shortcomings of one harness may not be visible until you contrast it with another that does the thing correctly.
My favourite is Claude Code's /rewind feature, which gets so many things wrong for having such a simple UI. (Does selecting the default option still wipe your filesystem changes by default when going back to a previous entry? I found a poorly-documented way to circumvent that "feature", which only adds to my initial point.)
Maybe they're actually using the model to review and clean up the code too? I hope you realize that "just merge whatever the LLM spits out on its first attempt" is not the only way to use LLMs.
Theyre usable but leave alot to be desired. SOme other tools like opencode seem sometimes to have less feature (if exact numbers) but at least always work.
Somehow with Claude and some others, random simple thigns dont work. Esp long solved problems like TUI rendering, special chars, breaking changes every release etc.
Surely as always with software this is my experience and might be not yours.
PS: Ive used IntelliJ since... always?. It was quite decent. But these last years it is just so diferent, somehow way less quality, so many random bugs that Im afraid theyre doing AI agentic coding which sucks for such an important tool.
IDEs should be build from real professional crafstman with an exact vision in their heads, not some wild AI.
Im starting to think in a year or two "We dont use AI in developing our products" will be some kind of marketing trick since people will start looking for that.
They spent a year trying to make a TUI renderer that doesn't flicker. They finally shipped a replacement, and there are still baffling bugs in it three months later. Like the difference between what it renders when you paste something into the input box, and what it shows after submit
I mean, I've mostly switched from Claude to Codex because I find the harness so buggy and annoying that I avoid it when I can. So.. as an (almost former) customer I don't think they're avoiding the issue at all.
Honestly it does not matter provided u develop tests that show success. D u really care how its coded if it is amazingly fast efficient and solves the problem? Maybe in some projects but not all.
We write code faster. It's not getting merged faster because we usually find something janky.
Decisions aren't made faster.
Understanding doesn't come faster.
Code doesn't have value. A well working system has value, the bottleneck is now between having code and having a working system, rather than producing the code.
With AI you can write a new Harry Potter novel in under an hour. But if I ask you about the story you can only answer “It involves wizards”.
In pre-AI era you could solve a bug in one week. But you would end up having a detailed mental picture of the system, which requires time to be created. That mental investment helps you make better decisions on the system or solve related problems instantly.
You dont have to review the code, but someone does. If it is a well-tuned adverserial reviewers, that would be more than sufficient. We are building zeroshot to let you customize that as a fixed graph for your repo, feel free to check it out!
reply