Like many, I became a SWE because I loved writing code. Ruby, in particular (whatever one thinks of its other strengths and weaknesses) can be very expressive and beautiful.
However, most code is corporate CRUD stuff. The world sort of runs on CRUD. But it's rarely particularly fun, or beautiful, or fulfilling. It tends to be pretty ugly because there's rarely time to make it beautiful. So I'm not losing much joy there.
Additionally, even with an agentic workflow, my team still thinks deeply about architecture issues. We believe that well structured code is better for bots and humans alike. So I am still exercising my engineering brain; I'm writing blueprints and supervising the bricklayers now instead of laying bricks.
Does that match your teams' experience at all? Sometimes I feel like folks on these threads are working in a parallel universe where the majority of coding tasks and somehow fulfilling and fun and not 99% tweaking reports, trying to implement arcane and contradictory business logic, or fighting with shitty and frustrating externalities. Maybe people found joy in those aspects. I guess there are all types of people in the world. I mean, I'm a sicko who loves writing SQL, something that most coders seem to hate.
I've been direct with my team about it being a legitimate
feeling of grief and the grieving process that comes along
with it. It's been helpful for folks to hear that they aren't
alone in that feeling.
This is super great. God, I wish all leaders had half this much empathy! Good on you.
I am one of those weirdos that found joy in business software. The clear specifications, reasonable timelines, and enough technical freedom, let you focus on clean, secure, resilient architectures.
I find that I struggle more with personal projects, as I tend to take things where I'm not even in the skirts of the learning curve, and consequently redo my work a few times.
> The clear specifications, reasonable timelines, and enough technical freedom, let you focus on clean, secure, resilient architectures.
I have no idea where you work but 90% of companies are not like that. In fact, it's the opposite: filled with awful management, terrible timelines and legacy code.
Engineering had always been a nuisance to them and getting basic approval for maintenance regarding technical debt has been an uphill battle.
> The clear specifications, reasonable timelines, and enough technical freedom, let you focus on clean, secure, resilient architectures.
The nicest thing was when you nailed down the architecture and then you are mostly twiddling your thumbs while waiting for the PM guys to have the next feature idea. Maintenance and bug fixing is a breeze.
Wow. That was hardly ever the case in my experience in ~30 years in the industry. I enjoy(ed) what you’re describing too but… how common was this for you?
Different person but I’ve been a SWE about ten years now - my favorite work has been improving the CRUD stuff people have left behind. Like the time I took a proprietary document they were using to document their REST apis and wrote a converter to turn those into OpenAI spec compliant yaml files, taking a single tenant application and making it a secure multienant application so instead of 80 different app services we had two, coalescing all their data sources into one normalized DB. There’s a lot of work I’ve done I’m really proud of. I’ve always found myself being the “in the muck” guy fixing up and bettering legacy code, I’ve always felt that was really rewarding and interesting.
Thats badass! I like doing that kind of work, but I got very jaded about it over the years, because management generally acted as if the engineers took a shit on the conference table in the middle of a Zoom call with their most important customer whenever anybody attempted to address technical debt, as opposed to delivering more more more features. Unless it was billable agency work, in which case they’d hate it even more.
With massive amounts of effort and strong communication it was usually possible to get some tech debt time, but management acted like they were doing YOU a huge favor. It was kind of a career killer, generally.
Alternatively it was possible to get this kind of work done on my own time… but that’s a marriage killer instead of a career killer.
Sure. If you get pleasure in fixing really badly broken shite, there’s tons of opportunity for that, but no money in it. Only money in new features, which is why the codebase is what it is. I have, on rare occasion, been the fixit guy called in temporarily to fix another teams work or get it out the door: that was fun. Short lived , but fun.
It's a shame, because there is money in it. It's just (like most sensible preventative maintenance) that the financial benefits are typically delayed. And when it comes to software, they're notoriously impossible to quantify.
Generally the only way to get it done in a corpo environment (and receive any kind of thanks or glory, as opposed to just torpedoing your own career) is to let things get so bad that production stability suffers and/or the team's ability to ship features is severely impacted.
...I realize I'm basically just echoing what you said, except more verbose. I'm venting!
The fun thing is: I don't think this will change that much with AI in the mix ;)
Shitty devs without AI are slowly building BS that wants to be cleaned up.
Shitty devs with AI are much more fasterly building BS that wants to be cleaned up.
I have a similar experience as yours, just add another 10-ish years to the timeline. Literally my first job out of uni, I got to work on a legacy code base that was another 15 years older than that which had a ton of reports all built differently over time with slight variations in both code and end user output that really all wanted to be "the same" both on the code and end user side. So I slowly but surely, one user/PM requested change at a time would build out a common layer of abstractions and shared code that both made the code shared and easier to maintain (great for me!) and consistent for users (great for the PM/users).
Heck, I built a rudimentary TransactSQL parser that'd output stored procedure call graphs to graphviz and then I printed it and hung it on the wall to get through all this and having an LLM to just make a mermaid out of this in 5-ish minutes would definitively have been faster than I was ;)
With an LLM all of that would just have been faster to do but still worthwhile and useful on both levels. I.e. both code abstraction/reuse and PM/user level. Similarly there was a report that one of the devs had to actually run manually and it took him a full week, so users that wanted the report would only request it once a year-ish. When I was asked to learn how to run it from him I was appalled and just took that week to put it into a script that we ran and emailed the report out once a month automatically. They ran that report manually like that for like 10 years prior to that :mindblown:
None of this is an "AI thing". It's a good dev vs. not so good dev thing. And none of it really has to do with coding as in "writing source code" as much as "solving problems for humans" somehow and the "language" you do that in changes over time. If you're the guy that's tied up with the particular language: yeah LLMs are gonna eat your lunch. If you're the problem solver type that uses whatever tools they have available, you're still in demand.
About 80% of the time. It wasn't all about setting a new foundation, even making the existing one solid enough so you can have time to refactor the cruft was enjoyable. I had more trouble in my youth, as I didn't understand why the business didn't change to accommodate my 'masterpieces', and I'd like to think I started enjoying it more as I became wiser.
Disclaimer: I had to take an opportunity at management in cybersecurity, and haven't coded in about five years. It helps a lot when talking to developers about risk mitigation, though.
One of the fun things to do would be to use the 8 hrs of work to spend about 4-5 hrs working on deliverables, which was usually sufficient to exceed expectations, using 2-3 of the remaining hours (I’d keep 1-2 hrs for myself), to make significant architectural improvements.
I’d bring in an improvement every 6 months or so and suddenly the app would get significantly better and I’d get credit for it.
Sorry for the delay. Mostly midrange (AS/400) consulting at manufacturing plants, then full time jobs at a heavy machine dealership, and a bank. Changed to PC development after about 15 years of that, then management, and it hasn't been the same after that, ha ha.
I think the technical freedom plays a big part, not because you're chasing the shiniest, but because you're choosing the tools that will hopefully outlast you. One of my former employers was 'modernizing' their IBM i (former AS/400) systems by porting them to Java and Windows Servers, and they were having all kinds of problems. Couldn't even get the EBCDIC to Unicode conversion quite right. I was lucky not to be there anymore.
I smile whenever I see EBCDIC, probably because I never had to deal with it.
But I'm sorta old and some of my school's texts were outdated so they had programming examples and explanatory text that treated EBCDIC on equal footing with ASCII.
So unlike anybody who dealt with EBCDIC conversions in the wild, seeing that acronym actually takes me back to happier days!
> not 99% tweaking reports, trying to implement arcane and contradictory business logic, or fighting with shitty and frustrating externalities.
Those are the parts the LLM is bad at. The 1% of stuff that was interesting was worth looking forward to.
I’m unconvinced about architecture mattering much. I used to think that, but these days I’m leaning towards it being the product of a problem we don’t have anymore. Architecture used to be dangerous and time-consuming to redo. If an AI can do that safely for $100 or $1000 in tokens (one day), that ROI is maybe negative
In a big production codebase with lots of active customers and data, architecture changes still feel pretty difficult even with LLMs accelerating them by an order of magnitude — migrating and backfilling data and all of that. That’s more complex than pointing Claude at a repo and saying “rewrite this in Rust.”
I guess you’re right, though - what is the ROI of keeping the architecture clean? It certainly saves time and money later, but it’s unclear if it’s a net win.
(It’s of course also significant to note that the upfront work of figuring out the right architecture and keeping the code clean has also been correspondingly reduced as well. Both sides of the equation are affected…)
The industry never really figured out a way to reliably answer this question in the pre-LLM days either, so… I suspect it will continue to be hard to determine.
in my experience it takes more than a day to sufficiently migrate bad architecture to a good one. like, many more days than it took to vibe the original implementation. having the prototype is good to understand what you want, but then going back and rebuilding it properly is a longer task. which is fine.
I think this is not "parallel universe". Selection bias: people who love to code often choose non trivial frontier work. They tend to grirf about agentic workflow the most. Myself included.
Yeah. “Just cut bullshitters out of your life” is about as unhelpful as “Don’t get sick!” when it comes to medical advice.
Like, duh yeah that’s the goal. Do that as much as possible, but a lot of your life is going to involve dealing with these people.
On a slightly deeper level, essentially nobody is 0% or 100% bullshitter. Nearly all of humanity falls into the middle. Even the most literal, level 2+ ASD folks I’ve known were, at a minimum, cagey at times.
No it's good advice. Most people don't react at all when they discover they were being bullshitted. Dropping interactions to the minimum you're forced to deal with them is still a huge improvement.
I think that’s a great reason to get into the psychology of bullshitters, or at least get really familiar with their tactics. You are going to have a lot of situations in your life when you can’t just flip a switch and get rid of them.
Most people, even toxic people, do have positive qualities.
At the most basic strategic level, it’s vital to recognize this so that we don’t fall for such people in the future. It’s sooooo easy to say “well so and so can’t be too toxic… I mean, they volunteer at the soup kitchen on weekends” or whatever. That’s part of how they (intentionally or unintentionally) lure you in. All very valuable for engineer-types, who may tend to be very literal in actions and their interpretations of others’ actions.
At a more nuanced strategic level, it’s good to recognize that since all people are a mix of positive and negative traits, How can we coexist with such people in cases when we can’t avoid it? It’s simplistic to say “just avoid toxic people” — it’s a good “north star” but is often impossible or impractical. e.g. it’s convenient to say “just cut toxic family members out of your life” and sometimes it’s the right call, but I’ve heard people toss that phrase around like it’s nothing and wow it’s heartbreakingly glib.
I think it’s also valuable for introspective purposes. It’s very easy to fall into the mindset of “well, I’m a good person because of X, Y, and Z” while blinding ourselves to negative ways in which we affect others.
Finally, yeah, sure — I admit it. I like a puzzle. Specifically the puzzle of figuring out if these people believe their own bullshit!
(For the record, I thought this movie was incredible, though HN’s collective “meh” on this thread was expected)
I found it a bit tough how many people in this thread are able to instantly spot people who are “bad” or “suck” and immediately cease all interaction and cut them out of their lives completely.
We must be living in very different worlds; in my experience these things are never that black&white.
I do agree that it’s very important to know where your boundaries are in life and stand firm on them; and you don’t need to keep people in your life who are a bad influence.
It’s just not as easy to make that distinction as some of the comments itt make it out to be …
It's important to be able to recognise behavioural patterns that suggest a narcissistic personality. Once you've established that, you need accept they won't change, because personality doesn't change, it's a stable pattern. Some people can spot these patterns due to long life experiences and know cutting out is the only option that leads to growth.
> you need accept they won't change, because personality doesn't change
I don’t think that is true. I know from myself and many of my friends or family that personalities can change, at times quite dramatically. Sometimes for “good” other times not so much.
The generally accepted view in psychology is that personality does not change over time. That is partly how it is defined. It's the part of us that doesn't change in adulthood.
Yes, people can change their behaviour. But those changes are not occurring at the personality level.
I haven't misunderstood anything. We are talking about narcissistic personality styles. They don't change. Yes, healthier adults have some flexibility, and can grow with experience. Narcissistic personalities are rigid. They can't self reflect. Whether you call it relative stability or stasis doesn't matter in the context of thia discussion. A narcissistic person is not going to have a watershed moment in response to a life crisis and all of a sudden decide to take accountability for their bad behaviour. They will continue to falsely identify other people as the problem, because it conflicts with their false sense of self and grandiosity.
You mean people that have narcissistic personality disorder (NPD) or people that exhibit narcissistic traits?
You previously wrote “personality doesn't change, it's a stable pattern” implying personalities not limited to narcissists.
If we are talking about people with severe or clinically elevated narcissism - sure it’s much harder to change since it’s basically a mental illness; but there are also many cases where people suffering from NPD improved their condition.
Which means they can change.
It might not be worth to you personally to be the facilitator of that change and that is fair, no one said that.
But saying that peoples personalities cant change as a blanket statement is just objectively wrong
We're talking specifically about bullshitting/lying/fraud, not other kinds of badness. The reason you have to cut such people out immediately once you realize what they're doing is because if you don't, they will keep trying to manipulate you. And usually if you catch someone in a lie it's pretty unambiguous: it really is black and white.
And I agree with the OP. In the past I've often been amazed and appalled at how someone can be proven to have lied, and the people around them barely update at all. They just brush it off and keep going back to be lied to again. It feels like the average normie has an extremely high tolerance for lying to the extent they don't feel it's much of a crime at all. Which I guess is why people like Elizabeth Holmes can thrive.
I assume you mean huge lies like Holmes-level lies.
Because everyone lies, it’s just a part of human nature.
Of course some lies are unforgivable. But it would need to be a pretty big lie for me to go and immediately cut people off.
Also I would argue the people that are obvious in their lying and manipulating aren’t the dangerous ones - those that aren’t obvious, that you don’t spot easily off the cuff are much more dangerous. And I’m 100% certain you also have such people in your life, it’s just unavoidable. And others might look at you and have the same thoughts: “look at this normie who doesn’t realise they are being lied to”
Overall I think there is a lot more nuance to this than you and OP make it out to be.
But I do understand the appeal of simple black/white narratives;
they are very attractive.
I just hope you treat the people close to you more merciful and forgiving.
> Also I would argue the people that are obvious in their lying and manipulating aren’t the dangerous ones - those that aren’t obvious, that you don’t spot easily off the cuff are much more dangerous.
Yeah. The person that's constantly bragging about fantasy stuff or reflexively claiming nothing was ever their fault might warrant cutting off because they're tiresome, but they're not likely to undermine you never mind pull off this level of scam because they're so unconvincing, and that might make them more harmlessly amusing than anything else. The manipulative person who's constant spreading false rumours might raise red flags, and definitely isn't someone you want to spend time with once you've spotted that some of their claims simply don't line up. But a large part of why people are often good at covering stuff up even if its the most heinous shit is that they manage to be plausible and likeable most of the time and generally tell the truth on other subjects.
Holmes likely fell into that category, though she didn't exactly downplay her lies in the way someone covering their deep dark secret usually does.
And yeah, in practice whether you cut someone off after learning their deep dark secret probably depends a lot on what it is and how it relates to you (and usually it's more "I'm hiding my affair" than "my billion dollar enterprise supplies fraudulent medical tests")
I mean objective lies. I've encountered people who lie about small unimportant things as a matter of course, yet they aren't merely exaggerating or misunderstanding. They are just objective, black/white lies, like claiming an important document says something when it doesn't even mention the topic at all.
It is not about "what you miss" by avoiding "toxic" people like Elizabet Holmes. Bad people like her are dangerous! She killed several people, by providing fake medical tests!
> like a puzzle... people believe their own bullshit!
Architecture preferences are a great example of things that we need to tell the LLMs.
We don't need to tell the LLM where the User model is. The LLM can find it.
However, IME we need to tell the LLM preferences like, "the User model should not have dependencies on other models" (you know, to avoid the God Model problem...)
To discover that preference, the LLM would (best case scenario) need to look at every model, and notice that the User model has no dependencies, but all the other models do, and infer that this is an architecture requirement. Except, some of our other leaf models might not have dependencies anyway, so it wouldn't necessarily be apparent. Or maybe it's a new project and none of the models have dependencies in which case our preference could not be deduced. Etc. All of this is token-intensive and unreliable.
(Perhaps even more ideally than placing this instruction in AGENTS.md or similar, it could be an inline comment in the User model itself. However, I have found LLMs to be very hit-or-miss when it comes to recognizing directives inside inline comments. And some architecture rules can't be easily expressed via inline comments in a particular file...)
Why would I put details about a singular class/model into the AGENTS.md?
Like you said, an inline comment probably makes more sense. Sure, maybe it misses it, then you can just tell it. Better than putting such granular details in a CLAUDE/AGENTS.md and better than having disparate markdown files or decisions documents that get unwieldy.
Quick, non-rhetorical sanity check, since you advocate passionately for just typing the same things into every session over and over.
You know the contents of AGENTS.md/CLAUDE.md just get injected into your context window by the harness? And that it's functionally equivalent to typing the same thing into every session?
There's no difference between typing "keep dependencies out of User" at the beginning of every session, and putting that into AGENTS.md, other than wasting a bunch of time.
Sure, maybe it misses it, then you can just tell it.
Re-doing that work after it gets things wrong is going to take roughly 1-3 orders of magnitude more tokens than including it in AGENTS.md/CLAUDE.md.
Here's a good concrete example. I've caught Claude writing python scripts to parse JSON instead of simply using `jq`. I initially attempted to correct this by adding `jq` to my allow list, but it still always reached for Python scripting first, before eventually figuring out it could use `jq`. You think that burned a few extra tokens vs. just telling it `use jq to parse JSON` in AGENTS.md?
You sure are patronizing for someone offended by me saying they're pulling stats out of their ass.
I don't really care how claude parses json. I'd say you're just wasting your time caring about that detail. I definitely wouldn't put that kind of useless info into my claude.md/agents.md.
If I DID care, yeah, I think it makes more sense to just correct the action. I'm not parsing json in every session so why would I want it in context for every session AND every sub-agent's session?
A lot of APIs we use return JSON - Github, Semaphore, Context7, Docker, etc. We're parsing JSON dozens of times per session. I think 5-6 tokens telling it to use `jq` in AGENTS.md was the right call for us.
Yeah. Basically IMO/IME the current best practice is to start the LLMs out with minimal/no skills/instructions/docs.
Notice the things they struggle with and the small problems (typically, environment issues IME) they repeatedly encounter and re-solve across multiple sessions. That is what your instructions should cover. When possible, move those instructions into skills, so they get loaded into context selectively instead of on every session. (Example: instructions for running specs, placed into a skill that only gets loaded into context when it’s time to run specs)
Again, this can be automated by the LLMs themselves: both Codex and Claude (and I’m assuming other major harnesses) know how to read their own transcripts and are good at looking for repeated friction and making concrete suggestions to reduce that friction in the future.
It takes a bit of a time investment on the user’s part, and every now and then you probably should throw it all out and start fresh so that the new batch of instructions can be appropriate for the current state of the repo and the capabilities of whatever model(s) you’re using.
Honestly, don't agree. I think even this is a waste of time and context.
The LLMs will always have these suggestions on things that will "reduce friction in the future" but I really think the LLM is a sycophant glazing you. Because even when you add those skills or prompts at some point the LLM ignores it or does whatever it wants to do anyways. Better to leave most of that out and deal with it when it arises.
Even your example, instructions for running specs... none of the current frontier models need this at all.
I think they're using Skynet. They seem to be claiming that part of the reason for not giving the LLM directions is because the LLM won't follow them anyway.
That must be a remarkable model. Smart enough to flawlessly discover everything on its own in every session... but also it just straight up just doesn't obey direction.
I'm saying a lot of devs/engineers little rain dances aren't really bringing the rain.
Keep a basic CLAUDE.md/AGENTS.md that keeps high level details. Have a way to reference other projects/apps/codebases to use as reference architecture.
Don't keep tons of markdown files. Don't keep "decision" documents (these are probably the worst). Don't make skills for every little thing (superpowers are dead these days, stop using them). Oh yeah, don't have the LLM generate docs for reference later... if it can generate the docs it doesn't really need them.
Nope. Opus and Sol for the most part. Haven't even used Astra or Fable.
At work it's all Anthropic. I spent a while on Sonnet models but have been pretty consistent about using Opus now. For personal projects I bounce between Sol/Terra and Gemini Flash.
Most of the models are good enough now. You don't need all these documents, memory, plugins, skills, etc. MCPs are good for hooking up to external sources of data (at work we have gitlab mcp, datadog mcp, and our knowledge base mcp... which the knowledge base is mostly worthless now because of all this AI generated documentation, but I digress). I DO still use beads on a lot of projects, but not always. My prompts these days are lazy as fuck: "review this repo, check out this other repo with architectural patterns we should follow and libraries/terraform/etc we should use, . What do you think? Let's review everything and discuss before we start building"
Oh no, I might have to stop the llm agent on occasion, or review what they did and tell them to fix a couple things. Way better use of tokens and my time than creating some rube goldberg machine. Way better than keeping a giant decision file that goes stale and causes context corruption because the llm only saw the OLD decision and not the UPDATED decision.
I understand what you're railing against in general: engineers who make these big memory/doc systems that quickly go stale and are at best useless and at worst actively harmful. Even more insidiously, these engineers are blind to this fact because maybe some of this stuff was effective 6-12 months ago and they haven't reassessed their workflows. Now that's dogma. We really do agree on that.
However. At least in the projects I've worked on, it's trivial to read the transcripts and notice that without enough direction, there are certain things the agents will burn lots of tokens to rediscover in every session. That's absolutely not dogma.
I think there's a balance between the tokens burned up rediscovering things vs tokens being used keeping docs/decisions/skills/etc.
And I think a lot of places are burning way more on the latter than the former. I think the former is more cost effective due to caching as well. So I HEAVILY lean towards the former especially with how effective I think it is these days.
I think there's a balance between the tokens
burned up rediscovering things vs tokens being
used keeping docs/decisions/skills/etc.
This mystical balance isn't difficult to measure. You don't have to believe anybody; just read the darn session transcripts. It's going to take at least one (but usually multiple) turns of grepping or other exploration to rediscover a thing. How many tokens does this add to your context? Since 1 token = ~4 characters you can count characters in your text editor.
Is this less, or greater than the number of tokens it would cost to simply add that thing to AGENTS.md?
If you're burning 5000 tokens to rediscover the same thing in every session, and it would take only 500 tokens to just put the thing in your AGENTS.md, it's an easy decision. Do you understand that paying 500 tokens in every session is less than paying 5000 tokens in every session?
If you're burning 5000 tokens to rediscover the same thing in 50% of your sessions, it's probably still worth it, but now you might consider placing it into a skill so that it only gets loaded into the 50% of sessions that need it.
If you're burning 5000 tokens to rediscover the same thing in 1% of your sessions then yeah, definitely skill. (You can add it to your prompts manually too, of course, but that is functionally equivalent to adding it to a skill that is set to manual invocation only...)
So I HEAVILY lean towards the former especially with
how effective I think it is these days.
How is this different from what I already said at the beginning of this conversation, where I recommended beginning with minimal (or no) skills or instructions... and then judiciously adding very lean ones based on actual observations? You claimed that this was a waste of time.
See here's the thing, you're pulling these numbers out of your ass when I'm making these recommendations from my observations that it's basically a wash and you get better results just having the agent read the code.
You aren't putting into 500 tokens what takes 5k tokens to discover. I've found that when I had these larger markdown files with all the details the models STILL went to the code to review. Then the markdown files get stale in spots, NO MATTER WHAT. Not only stale, but if you're having agents maintain these files they WILL hallucinate or input superfluous or unnecessary info. Now that I have short markdown files (or none in some instances) the agent just looks at the code and it's pretty simple to figure out.
And yes, I think most of the skills are a waste of time. You're just creating more shit to maintain. The most usable and reusable skills end up being short and concise, to where you could just type them into the prompt in any number of ways adhoc... effectively making the skill mostly worthless. The big skills end up being too much and less effective ime. And it can be difficult to rely on agents to invoke skills or the right skills.
If my skills need maintenance and hand holding, what am I really gaining?
As I optimistically hope you may be able to eventually understand with what I assume to be significant effort on your part[1], the numbers are going to be different on a case by case basis. Thus, my generalization.
This conversation is over, as you have been moving goalposts in various random directions: careening between "no directions for LLMs!" and then talking about your "minimal" AGENTS.md and markdown files, and then disagreeing with me for having minimal AGENTS.md and markdown files, in favor of your approach which involves... minimal AGENTS.md and markdown files, apparently. Luckily, it all sounds so schizo that at least nobody will take you seriously enough to follow your bad advice. It's exhausting talking to you, but surely not as exhausting as being you, so I do still have sympathy. Good luck.
---
[1] remember to hydrate, a good night's sleep helps too
In general, as LLMs get smarter, “less is more” becomes increasingly true. You’re polluting their context with dozens or hundreds of instructions, all of which the LLM tries to satisfy. However,
You don’t need documentation […or…] memory
…oh heck no. Easy to miss at Claude and Codex’s default detail level but if you read your actual session transcripts, you are almost certain to notice the LLM solving lots and lots of the same little problems over and over again.
The code IS the documentation
First, lots of things cannot be learned from the code.
Trivial example: LLMs struggled with QA on our app. They didn’t know how to find the seeded test accounts. They would create new ones and do it wrong, or find the seeded test accounts but not know their passwords because they were encrypted, so they’d change the passwords but not tell the other agents. Shitloads of tokens burned. It was a no-brainer to just give them the credentials in a skill that gets loaded when they do QA.
You could say that’s an environment issue, not a code issue. But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
The days of “you are the world’s greatest Python programmer, write good code” or whatever are over (if that style of prompting even worked in the first place) and as I said less is more. But, still….
> You could say that’s an environment issue, not a code issue
Yeah, that's exactly what I'd say. Or maybe you're just approaching the problem now.
You have to prompt these QA agents, correct? Why not give the instructions on where to get credentials in the original prompt?
> But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
I did say I'll keep an AGENTS.md or CLAUDE.md. I keep that SUPER high level. The most depth I'll give is info about example projects (That the llm can reach using an MCP) to follow for architectural patterns.
Why not give the instructions on where to get
credentials in the original prompt?
We don't push code unless it's been QA'd, and the LLMs need to know the credentials every time they do automated QA. It's very rare for me have a session in that repo where they wouldn't need that info. So it's a great candidate for AGENTS.md
Another concrete example would be codebase conventions. We prefer lean models. Cross-model concerns go into service objects. Without this direction LLMs tend to default to stuffing too much code into the models themselves, as is the de facto standard for most MVC apps and therefore this is how LLMs are trained.
Even if they could discover our large codebase's conventions flawlessly on their own in every session, this absolutely would require nontrivial work repeated in every session: quite a few turns grepping, conversing with LSPs, or whatever.
So yes... we could describe our conventions in every single session by typing it right into the prompts... right after typing the directions to find the test credentials... and the other ten or twelve things we'd be telling it every single time...
I'm assuming these QA agent are automated, probably in CI/CD. And I'm assuming the prompt that runs these QA agents is from a codebase in VC. Why would I put QA instructions into the codebase's repo as a top level markdown file and not in the pertinent part of my QA process?
Even with your documentation, the llm is gonna do a lot of that grepping and discovery. Unless you have such comprehensive documentation that it's basically code itself... in which case, it should just use the code as documentation.
The good news is that I don't think what you guys are doing is going to be really bad. It's just not near optimal and it's creating this weird dogma around all these "tools"
Do you not run your tests locally? Regardless of what's happening or not happening in CI/CD, I would think that most workflows involve running the tests locally. We do TDD more or less and for the browser-based tests, the LLMs need to know how to log into stuff.
No, these kind of tests aren't generally run locally. Not to get to prod at least. These are all deterministic tests to get to prod.
Locally? Our devs can do whatever they want. For anything I'm working on, for local deployments and testing, I generally build it out so it's set in Makefile/Taskfile. If you're running your determinstic tests locally it should really just be a single command that bootstraps everything and runs the tests, or updates a certain part of your app and runs the tests. The LLM shouldn't need credentials in that instance.
Regardless I'm deploying a local stack of some sort all the credentials and everything are probably stored locally or in an env var where the llm will have access. So if I was having the LLM drive a browser during active development, it can look at the files.
Sure, I guess we could put in the top level markdown more details about this... but why? It takes little time or context for it to figure it out. We have shit change so frequently that it's just something else we have to maintain.
Yikes. I'm not the poster you're responding to, but
1. In this absolutely wild era of model progress, with frontier and near frontier models leapfrogging themselves seemingly on a weekly basis it's good to get experience with a variety of providers
2. This isn't exactly hard with OpenRouter!
3. I've had very good results having models from one provider critique plans and code written by models from another provider. Can't do that without subs to at least two providers
I can't speak for parent poster, but it's not the cuteness. It's the "putting a cute face on something that's actually a bit sinister."
Hello Kitty isn't trying to harvest anybody's data or take anybody's job. Sanio makes cute things and they hope that you will exchange money for their goods. That's the whole relationship.
However, most code is corporate CRUD stuff. The world sort of runs on CRUD. But it's rarely particularly fun, or beautiful, or fulfilling. It tends to be pretty ugly because there's rarely time to make it beautiful. So I'm not losing much joy there.
Additionally, even with an agentic workflow, my team still thinks deeply about architecture issues. We believe that well structured code is better for bots and humans alike. So I am still exercising my engineering brain; I'm writing blueprints and supervising the bricklayers now instead of laying bricks.
Does that match your teams' experience at all? Sometimes I feel like folks on these threads are working in a parallel universe where the majority of coding tasks and somehow fulfilling and fun and not 99% tweaking reports, trying to implement arcane and contradictory business logic, or fighting with shitty and frustrating externalities. Maybe people found joy in those aspects. I guess there are all types of people in the world. I mean, I'm a sicko who loves writing SQL, something that most coders seem to hate.
This is super great. God, I wish all leaders had half this much empathy! Good on you.reply