I remember buying a lifetime VPS that was listed on LowEndBox for like $20. Can't remember the spec and I guess a lifetime means 5 years, but I couldn't complain for $20
Pre-llm I'd say it can be more undertaking than most people care to take to move to "serverless" or "cloud native" tech. But now it's probably worth taking a crack at it.
I don't miss sshing into servers or using puppet, ansible, salt, chef.
I don't use any cheap VPS providers. Everything is in AWS for me. I do run a single small ec2 instance ($8/mo?) that runs a personal mediawiki and a game server (for some multiplayer games I made).
I run a bunch of cheap (free?) custom apps for myself in the "serverless" or "cloud native" stuff in AWS. Everything else is on my NAS running on my home network (I see people mentioning qbittorrent, jellyfin, etc).
I stopped using cheap vps providers probably a decade ago, not a huge difference between a $5/mo cheap vps to a basic ec2 instance in pricing. And there were always issues with the cheap vps. Not to mention how easy it is to backup mediawiki + mysql in AWS vs having to spin up ANOTHER vps. Maybe there are cheap vps providers that offer object storage now, but I never bothered looking.
You don't need documentation or the 3rd party memory systems. The code IS the documentation.
All this stuff is LLM rube goldberg machines. It just pollutes context.
I barely use AGENTS.md/CLAUDE.md these days. And where they remain, it's super basic high level stuff.
I'm honestly still kicking myself in the ass on many projects where I did something similar to this. I kept tons of markdown docs and decision docs. Now those things are just causing problems because they got stale. Even after having sessions of reconciling documentation, the LLM just gets confused.
Haha, all the software devs who hate writing documentation are naturally finding their preexisting beliefs reinforced when the LLM is able to discern intent without docs. An LLM can be spooky impressive at reading minimized or obfuscated code, for example.
But this article argues that LLMs do better when the context is smaller — when it can understand the totality of the task with as little context as possible. And so having correct API-level docs is greatly advantageous. Anecdotally, this rings true to me — when the local context is good and clear, the LLM writes code matching my intent even when my prompt is sloppy and poorly specified.
Rejoice! The LLM will write the docs for you, relieving you of most of the work.
However without intervention, it will do too much and record absurdly verbose docs (similar to how an LLM will relentlessly refactor your code until you instruct it to move in minimal, incremental changesets). You will still need to edit down what the LLM generates.
Yeah. Basically IMO/IME the current best practice is to start the LLMs out with minimal/no skills/instructions/docs.
Notice the things they struggle with and the small problems (typically, environment issues IME) they repeatedly encounter and re-solve across multiple sessions. That is what your instructions should cover. When possible, move those instructions into skills, so they get loaded into context selectively instead of on every session. (Example: instructions for running specs, placed into a skill that only gets loaded into context when it’s time to run specs)
Again, this can be automated by the LLMs themselves: both Codex and Claude (and I’m assuming other major harnesses) know how to read their own transcripts and are good at looking for repeated friction and making concrete suggestions to reduce that friction in the future.
It takes a bit of a time investment on the user’s part, and every now and then you probably should throw it all out and start fresh so that the new batch of instructions can be appropriate for the current state of the repo and the capabilities of whatever model(s) you’re using.
Honestly, don't agree. I think even this is a waste of time and context.
The LLMs will always have these suggestions on things that will "reduce friction in the future" but I really think the LLM is a sycophant glazing you. Because even when you add those skills or prompts at some point the LLM ignores it or does whatever it wants to do anyways. Better to leave most of that out and deal with it when it arises.
Even your example, instructions for running specs... none of the current frontier models need this at all.
I think they're using Skynet. They seem to be claiming that part of the reason for not giving the LLM directions is because the LLM won't follow them anyway.
That must be a remarkable model. Smart enough to flawlessly discover everything on its own in every session... but also it just straight up just doesn't obey direction.
I'm saying a lot of devs/engineers little rain dances aren't really bringing the rain.
Keep a basic CLAUDE.md/AGENTS.md that keeps high level details. Have a way to reference other projects/apps/codebases to use as reference architecture.
Don't keep tons of markdown files. Don't keep "decision" documents (these are probably the worst). Don't make skills for every little thing (superpowers are dead these days, stop using them). Oh yeah, don't have the LLM generate docs for reference later... if it can generate the docs it doesn't really need them.
Nope. Opus and Sol for the most part. Haven't even used Astra or Fable.
At work it's all Anthropic. I spent a while on Sonnet models but have been pretty consistent about using Opus now. For personal projects I bounce between Sol/Terra and Gemini Flash.
Most of the models are good enough now. You don't need all these documents, memory, plugins, skills, etc. MCPs are good for hooking up to external sources of data (at work we have gitlab mcp, datadog mcp, and our knowledge base mcp... which the knowledge base is mostly worthless now because of all this AI generated documentation, but I digress). I DO still use beads on a lot of projects, but not always. My prompts these days are lazy as fuck: "review this repo, check out this other repo with architectural patterns we should follow and libraries/terraform/etc we should use, . What do you think? Let's review everything and discuss before we start building"
Oh no, I might have to stop the llm agent on occasion, or review what they did and tell them to fix a couple things. Way better use of tokens and my time than creating some rube goldberg machine. Way better than keeping a giant decision file that goes stale and causes context corruption because the llm only saw the OLD decision and not the UPDATED decision.
I understand what you're railing against in general: engineers who make these big memory/doc systems that quickly go stale and are at best useless and at worst actively harmful. Even more insidiously, these engineers are blind to this fact because maybe some of this stuff was effective 6-12 months ago and they haven't reassessed their workflows. Now that's dogma. We really do agree on that.
However. At least in the projects I've worked on, it's trivial to read the transcripts and notice that without enough direction, there are certain things the agents will burn lots of tokens to rediscover in every session. That's absolutely not dogma.
I think there's a balance between the tokens burned up rediscovering things vs tokens being used keeping docs/decisions/skills/etc.
And I think a lot of places are burning way more on the latter than the former. I think the former is more cost effective due to caching as well. So I HEAVILY lean towards the former especially with how effective I think it is these days.
I think there's a balance between the tokens
burned up rediscovering things vs tokens being
used keeping docs/decisions/skills/etc.
This mystical balance isn't difficult to measure. You don't have to believe anybody; just read the darn session transcripts. It's going to take at least one (but usually multiple) turns of grepping or other exploration to rediscover a thing. How many tokens does this add to your context? Since 1 token = ~4 characters you can count characters in your text editor.
Is this less, or greater than the number of tokens it would cost to simply add that thing to AGENTS.md?
If you're burning 5000 tokens to rediscover the same thing in every session, and it would take only 500 tokens to just put the thing in your AGENTS.md, it's an easy decision. Do you understand that paying 500 tokens in every session is less than paying 5000 tokens in every session?
If you're burning 5000 tokens to rediscover the same thing in 50% of your sessions, it's probably still worth it, but now you might consider placing it into a skill so that it only gets loaded into the 50% of sessions that need it.
If you're burning 5000 tokens to rediscover the same thing in 1% of your sessions then yeah, definitely skill. (You can add it to your prompts manually too, of course, but that is functionally equivalent to adding it to a skill that is set to manual invocation only...)
So I HEAVILY lean towards the former especially with
how effective I think it is these days.
How is this different from what I already said at the beginning of this conversation, where I recommended beginning with minimal (or no) skills or instructions... and then judiciously adding very lean ones based on actual observations? You claimed that this was a waste of time.
See here's the thing, you're pulling these numbers out of your ass when I'm making these recommendations from my observations that it's basically a wash and you get better results just having the agent read the code.
You aren't putting into 500 tokens what takes 5k tokens to discover. I've found that when I had these larger markdown files with all the details the models STILL went to the code to review. Then the markdown files get stale in spots, NO MATTER WHAT. Not only stale, but if you're having agents maintain these files they WILL hallucinate or input superfluous or unnecessary info. Now that I have short markdown files (or none in some instances) the agent just looks at the code and it's pretty simple to figure out.
And yes, I think most of the skills are a waste of time. You're just creating more shit to maintain. The most usable and reusable skills end up being short and concise, to where you could just type them into the prompt in any number of ways adhoc... effectively making the skill mostly worthless. The big skills end up being too much and less effective ime. And it can be difficult to rely on agents to invoke skills or the right skills.
If my skills need maintenance and hand holding, what am I really gaining?
As I optimistically hope you may be able to eventually understand with what I assume to be significant effort on your part[1], the numbers are going to be different on a case by case basis. Thus, my generalization.
This conversation is over, as you have been moving goalposts in various random directions: careening between "no directions for LLMs!" and then talking about your "minimal" AGENTS.md and markdown files, and then disagreeing with me for having minimal AGENTS.md and markdown files, in favor of your approach which involves... minimal AGENTS.md and markdown files, apparently. Luckily, it all sounds so schizo that at least nobody will take you seriously enough to follow your bad advice. It's exhausting talking to you, but surely not as exhausting as being you, so I do still have sympathy. Good luck.
---
[1] remember to hydrate, a good night's sleep helps too
100% agree. Most of the things that make development better for humans also make development better for agents… and I think docs are even more important with agents, because of some kind of multiplicative effect. The agents are coding faster, and the benefits of documentation are somewhat more pronounced because of the speed.
You guys show me a measurable difference on something with docs vs without docs and I'll believe it.
The docs just make you feel good. They're not worth anything. They just create more work if anything because now you don't only have to fight entropy in your codebase but also your docs.
Uh, you aren't providing numbers either, just appending the same assertion to the end of each branch of the discussion. The result is the same as when the LLM does that: geometric increase in what we have to fight through to reason about the problem.
With the LLM, though, I have some influence: I can instruct it to be more succinct. I don't have a specific top-level instruction about that at present in any global prefs file, but I've occasionally asked it to summarize and archive and it's done an okayish job of pruning.
Yes, you're right, I'm not providing numbers. But that's because I'm just using the tool as designed... the vanilla version. I'm using the baseline. I believe anthropic and openai publish plenty for my side of the argument already.
I'm not the one bolting things on claiming it increases productivity. Why would I ADD stuff without any proof or evidence that it works? I'm simply NOT adding these things.
"CLAUDE.md files solve this by giving Claude persistent context about your project. A well-configured CLAUDE.md transforms how Claude works with your specific project. The file serves multiple purposes: providing architectural context, establishing workflows, and connecting Claude to your development tools. Each addition should solve a real problem you have encountered, not theoretical concerns about what Claude might need."
I pretty clearly say I still use CLAUDE.md/AGENTS.md
I've just been trimming down to the recommended < 100 lines and I keep only super high level info in there and allow the agents to figure things out by reading the code.
I wouldn't use a blog post from Nov 2025 be my guiding light.
Anthropic over the past few months have said they've cut their system prompts 80%. These are current best practices. The models are good enough and can figure things out on their own BETTER than you can present the info in markdown files and "decision" documents.
The docs end up stale or you waste a ton of time and tokens keeping them up to date. And you can't even trust the LLMs to keep the docs up to date, you will still need to review it yourself keep a bunch of shit out of them.
Architecture preferences are a great example of things that we need to tell the LLMs.
We don't need to tell the LLM where the User model is. The LLM can find it.
However, IME we need to tell the LLM preferences like, "the User model should not have dependencies on other models" (you know, to avoid the God Model problem...)
To discover that preference, the LLM would (best case scenario) need to look at every model, and notice that the User model has no dependencies, but all the other models do, and infer that this is an architecture requirement. Except, some of our other leaf models might not have dependencies anyway, so it wouldn't necessarily be apparent. Or maybe it's a new project and none of the models have dependencies in which case our preference could not be deduced. Etc. All of this is token-intensive and unreliable.
(Perhaps even more ideally than placing this instruction in AGENTS.md or similar, it could be an inline comment in the User model itself. However, I have found LLMs to be very hit-or-miss when it comes to recognizing directives inside inline comments. And some architecture rules can't be easily expressed via inline comments in a particular file...)
Why would I put details about a singular class/model into the AGENTS.md?
Like you said, an inline comment probably makes more sense. Sure, maybe it misses it, then you can just tell it. Better than putting such granular details in a CLAUDE/AGENTS.md and better than having disparate markdown files or decisions documents that get unwieldy.
Quick, non-rhetorical sanity check, since you advocate passionately for just typing the same things into every session over and over.
You know the contents of AGENTS.md/CLAUDE.md just get injected into your context window by the harness? And that it's functionally equivalent to typing the same thing into every session?
There's no difference between typing "keep dependencies out of User" at the beginning of every session, and putting that into AGENTS.md, other than wasting a bunch of time.
Sure, maybe it misses it, then you can just tell it.
Re-doing that work after it gets things wrong is going to take roughly 1-3 orders of magnitude more tokens than including it in AGENTS.md/CLAUDE.md.
Here's a good concrete example. I've caught Claude writing python scripts to parse JSON instead of simply using `jq`. I initially attempted to correct this by adding `jq` to my allow list, but it still always reached for Python scripting first, before eventually figuring out it could use `jq`. You think that burned a few extra tokens vs. just telling it `use jq to parse JSON` in AGENTS.md?
You sure are patronizing for someone offended by me saying they're pulling stats out of their ass.
I don't really care how claude parses json. I'd say you're just wasting your time caring about that detail. I definitely wouldn't put that kind of useless info into my claude.md/agents.md.
If I DID care, yeah, I think it makes more sense to just correct the action. I'm not parsing json in every session so why would I want it in context for every session AND every sub-agent's session?
A lot of APIs we use return JSON - Github, Semaphore, Context7, Docker, etc. We're parsing JSON dozens of times per session. I think 5-6 tokens telling it to use `jq` in AGENTS.md was the right call for us.
"docs and maps" are hardly specifying preferences. I DID say I still often keep a high level AGENTS.md or CLAUDE.md. Where I have them, they're super high-level. In places where we have well defined and followed standards that top level markdown file can be as simple as "check out this other repo for patterns" and where I don't, the markdown file is high level overview of what the app is and then a high level overview of architecture or some preferences. Always a shorter/smaller doc.
I don't mind SOME documentation. I'm growing extremely frustrated with the absolute MOUNTAIN of docs, comments, decision documents, etc being created. I'm so fucking tired of the LLM rube goldberg machines being built.
Yeah, you seem to have the same thoughts on this to me. I think it applies to more than just the documentation, a lot of the skills/plugins/etc fall under this too (like almost all of them ime). It's all prose, so it's easy to commit and ship it. It makes engineers feel like they're generating value.
I've got one example that's at the top of my mind driving me crazy right now.
We have to move some git repos. It requires making sure there are no secrets in git history or committed, in a lot of cases we're just archiving, creating new repos, copying the code over. We just have to mostly mirror the config of the prior repo. There are probably about 100 repos that we need to move. I farmed this out to 3 of my ICs, a Sr and 2 jr/mid level guys. I thought one of them would figure out a way to mass move them and worst case they just move them real quick... shouldn't be too hard.
Well my Sr has spent the past 6-8 weeks building claude skills/plugins to move the repos. Every week he finds an edge case or the tool doesn't do something perfect, so every sprint he has "fixes" for it.
So far he's moved 5 repos. The jr/mid level guys, one has moved 30 the other has moved 10. I grabbed one of these tasks, threw claude at it (again, no tools or any bullshit), gave it a quick prompt, knocked out one of these in 20 minutes (while multi-tasking).
This same IC has built an MR review tool. It takes 20-60 minutes to run and costs $20-$50+ each run. Wants to put it into our CI/CD so that it runs on MRs. If you include all the pipelines that would run because of a dev updating a branch in an MR we're talking hundreds to thousands of pipelines a day.
I thought this dev was token maxing or just trying to look busy and I've had to sit down and talk to him, pull him aside for quick 1 on 1s, and to be honest, I don't get the feeling that he knows what he's doing is a gigantic waste of time. The LLMs are sycophants that glaze you. You think everything coming out of it is GREAT work because it tells you that. And I think this is happening across all layers of every org out there in various degrees.
Yeah I have similar struggles: I witnessed several multi-month projects that should never have been started. But they would sound cool on paper, and sycophantic LLMs would support any shit-on-a-stick, so they were kicked off, only to be killed months later. Huge boosts in productivity coming from coding agents are often negated by AI psychosis. But is some ways I enjoy it, because instead of being the great equalizer AI turns out to be the great differentiator.
Review tool is an interesting one - I find LLMs to be quite useful for code reviews, but people should really stop associating code reviews with Github/Gitlab and running them as a part of CI. Those platforms were built for humans because reviewing patches over email was a terrible experience. Clankers have no problem with raw patches, in fact they prefer them over fancy web UI. Code hosting platforms are not a good place for agentic code reviews, and they never will be.
Most LLM reviews should only run locally on developer's machine, in your regular coding harness.
The people in here sticking to their dogma has motivated me to start building out tests for a lot of this stuff. I spent a chunk of my day today testing the MR tool today.
* vanilla claude with our MCPs + a decent prompt (like a paragraph tops)
* an agent that was previously made, much more concise skills
* the mr tool this IC created
The MR tool didn't even catch some of the stuff in our domain that the others caught and it took 45minutes, cost $20 vs 4mins for vanilla (<$1) and 7mins for the agent ($3ish)
And yeah, code-reviews are supposed to be by humans, but we have a ton of reviews come in where there's a ton of low hanging fruit. This tool is meant to be a gate between getting an actual human code review and some of the slop coming in these days.
I'll go through a bunch more MR reviews, see how things go. But I've gotta start figuring out how to test the documentation, decision docs, and other plugins/skills.
Don't agree. LLM generated docs are some of the worst because like I said, the LLM never trims, only amends. So we used to do X but now we've had an architectural change or some type of change where we should never do X. Instead of just removing the instructions to do X in the docs, it amends them, "we made the decision to no longer do X because of Y". Now it has X multiple times in context instead of just not having X in context at all.
I hesitate to make blanket statements, since this is quickly evolving, but in general I agree. LLM generated docs will quickly degrade to overly verbose, unreadable crap.
There are some things that can't be derived from the code and old school PMP documents are suddenly useful again.
Who are your stakeholders? How are you going to communicate with them? What kinds of approvals do you need to do certain things? Etc. Human devs have long struggled with this stuff and LLMs aren't at better.
In general, as LLMs get smarter, “less is more” becomes increasingly true. You’re polluting their context with dozens or hundreds of instructions, all of which the LLM tries to satisfy. However,
You don’t need documentation […or…] memory
…oh heck no. Easy to miss at Claude and Codex’s default detail level but if you read your actual session transcripts, you are almost certain to notice the LLM solving lots and lots of the same little problems over and over again.
The code IS the documentation
First, lots of things cannot be learned from the code.
Trivial example: LLMs struggled with QA on our app. They didn’t know how to find the seeded test accounts. They would create new ones and do it wrong, or find the seeded test accounts but not know their passwords because they were encrypted, so they’d change the passwords but not tell the other agents. Shitloads of tokens burned. It was a no-brainer to just give them the credentials in a skill that gets loaded when they do QA.
You could say that’s an environment issue, not a code issue. But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
The days of “you are the world’s greatest Python programmer, write good code” or whatever are over (if that style of prompting even worked in the first place) and as I said less is more. But, still….
> You could say that’s an environment issue, not a code issue
Yeah, that's exactly what I'd say. Or maybe you're just approaching the problem now.
You have to prompt these QA agents, correct? Why not give the instructions on where to get credentials in the original prompt?
> But as far as actual code IME at a minimum we have to tell the agents our general repo structure and architecture patterns.
I did say I'll keep an AGENTS.md or CLAUDE.md. I keep that SUPER high level. The most depth I'll give is info about example projects (That the llm can reach using an MCP) to follow for architectural patterns.
Why not give the instructions on where to get
credentials in the original prompt?
We don't push code unless it's been QA'd, and the LLMs need to know the credentials every time they do automated QA. It's very rare for me have a session in that repo where they wouldn't need that info. So it's a great candidate for AGENTS.md
Another concrete example would be codebase conventions. We prefer lean models. Cross-model concerns go into service objects. Without this direction LLMs tend to default to stuffing too much code into the models themselves, as is the de facto standard for most MVC apps and therefore this is how LLMs are trained.
Even if they could discover our large codebase's conventions flawlessly on their own in every session, this absolutely would require nontrivial work repeated in every session: quite a few turns grepping, conversing with LSPs, or whatever.
So yes... we could describe our conventions in every single session by typing it right into the prompts... right after typing the directions to find the test credentials... and the other ten or twelve things we'd be telling it every single time...
I'm assuming these QA agent are automated, probably in CI/CD. And I'm assuming the prompt that runs these QA agents is from a codebase in VC. Why would I put QA instructions into the codebase's repo as a top level markdown file and not in the pertinent part of my QA process?
Even with your documentation, the llm is gonna do a lot of that grepping and discovery. Unless you have such comprehensive documentation that it's basically code itself... in which case, it should just use the code as documentation.
The good news is that I don't think what you guys are doing is going to be really bad. It's just not near optimal and it's creating this weird dogma around all these "tools"
Do you not run your tests locally? Regardless of what's happening or not happening in CI/CD, I would think that most workflows involve running the tests locally. We do TDD more or less and for the browser-based tests, the LLMs need to know how to log into stuff.
No, these kind of tests aren't generally run locally. Not to get to prod at least. These are all deterministic tests to get to prod.
Locally? Our devs can do whatever they want. For anything I'm working on, for local deployments and testing, I generally build it out so it's set in Makefile/Taskfile. If you're running your determinstic tests locally it should really just be a single command that bootstraps everything and runs the tests, or updates a certain part of your app and runs the tests. The LLM shouldn't need credentials in that instance.
Regardless I'm deploying a local stack of some sort all the credentials and everything are probably stored locally or in an env var where the llm will have access. So if I was having the LLM drive a browser during active development, it can look at the files.
Sure, I guess we could put in the top level markdown more details about this... but why? It takes little time or context for it to figure it out. We have shit change so frequently that it's just something else we have to maintain.
Code as documentation works better when the code is declarative or a DSL. These capture intent and promises (as in promise theory) better.
When it is not, it has to be reasoned out and does not work well for documentation.
Other things that code and tests alone do not capture well:
- promises (as in Promise Theory) made to other parties. Claude already has PT in its training data.
- Constraints-inducing-properites, as in Roy Fielding / Christopher Alexander. While tests, and property testing can capture properties, there is no formal connection to the constraints that induces fhem. By constraints, I am not talking about business requirements and business value — those are better understood through Promise Theory. I am talking about things like at-least-once delivery or total ordering (from append-only constraint). Claude already has Fielding’s dissertation and Alexander’s works and ideas in its training data.
- grammars, as in pattern panguages (not just patterns) a la Alexander / Fielding are also not captured in code alone. These tell both humans ans AI how to extend a pattern, and how to identify anti-patterns (when they violate a constraint-inducing-property)
- LLMs are trained with many different worldviews and bounded contexts at the same time, and is very capable of translating across it. However, these need to be soelled out, otherwise it would talk in whatever it infers
Specifications written for the exact way components are wited together run into that stale doc problem. Although it takes much more human attention and token burn to describe things in terms of pattern language and promise theory, it becomes easier over time. The actual implementation plan tends to fall out more cleanly when all those other stuff are at least considered. This is where I have been spending most of my time.
I liked this advice when humans wrote code. Though even then I'd urge people to write meaningful commit messages that capture the "why" of what they did, so no one tramples their intent by mistake.
But not sure it works in an age where most code is LLM-generated. Especially if that code is not even reviewed by humans (irresponsible or not, it's happening), and commit messages are also generated by AI. I think something is needed to separate "what did the human operator intend" from what the agent went and built.
I do agree that this gets way overengineered. My approach has been more or less what you stopped doing though - committing all our timestamped "plan/implementation docs" and "investigation docs" that document what the user wanted + empirical findings, and making all prior session transcripts searchable. It's seemed mostly helpful? For whatever reason I haven't run into many staleness problems so far.
I'm mainly aiming my frustrations at all the markdown files being committed, all the additions to knowledge bases, all the comments in the code (especially the ones referencing specific JIRA tickets). This stuff isn't helpful, it gets hella stale. I've had the LLM fuck up plenty due to these docs and comments.
Commit message ARE EXACTLY where architectural decisions or nuance should go. Not another fucking .md or more comments.
Unfortunately, that is not how many software development projects work. A customer may request certain things, or the software might be part of a larger system.
Consider working on software for a coffee machine. Why is pin 42 (GRIND) activated every now and then? Would you really want to document that in Git commit messages?
Yeah, I'm going too far the other way. There are definitely legitimate reasons to comment in code and have documentation. I just feel like we're seeing a MASSIVE amount of documentation now and it's so much that it's mostly worthless.
I'm seeing decision files that are so big the LLM can't fit it all in context on some projects. Then the LLM makes decisions that revert previous ones and later sessions don't pick that up so it sticks to the original decision. Now in some sessions, every so often I have to remind the LLM, "no, we changed that later, we do it this way now"
And I'm seeing our knowledge base grow to a completely useless giant mess of stale, outdated, duplicated, or superfluous info. LLMs often pull unrelated info or confuse similar but different things or get old documentation for something that's been updated to new documentation in a different part of the knowledge base. And these are LLMs generating the docs. And we have LLMs and agents reconciling. But it doesn't seem to always get everything.
For code comments, it's terrible because the comments are starting to get larger than the code. A large chunk of the comment can be discerned from the code itself. Then the comment has details on why that maybe don't quite make much sense. It's like the LLMs start using words in a specific context that doesn't really apply to the word in normal spoken english. Then it will also often include a specific JIRA ticket id, you check the JIRA ticket, you see that yeah, the code was changed because of that JIRA ticket, but it's not really related to the ticket itself, it was just a blocker. But now the comment forever links it to THAT ticket (And then now sometimes the LLM pulls in that ticket with the atlassian mcp).
It’s a weird thing isn’t it, the urge to save these artifacts? The worst is when the llm refers to the decision and design in code comments. In my opinion there’s one thing that is worth documenting; tricky architecture or implementation details that are some how counterintuitive to what would have normally been done. But again this can be documented in the code and tests.
I've never gotten a lot of mileage out of treating tests as documentation. It is getting worse with how many people just tell the LLM to produce tests matching the code.
I don't understand. Code doesn't capture the context in which decisions were taken: why is code the way it is? What is important? What is not? How can agents make correct decisions without knowing context that cannot be inferred from code?
> If it's not, that's what commit messages are for.
Wouldn't this require that your commits would have to be as frequent as each new reasons why? That could easily end up being a commit per function, which seems crazy to me.
I always put any important why close to the code that I'm explaining. For architectural explanations, maybe at the top of the file, or for really high level "how it all works together", in an md.
I'm seeing ENORMOUS decision files and when I read them a lot of the decisions are self-evident. Or the decisions flip flop but the old decisions remain in the file, which fucks up context. It's often better to completely remove a reference to X rather than leave it and have an amendment later that says "don't do X". Because X is in context, now the LLM is more likely to stumble into doing X.
There's always been vendor lock-in. Someone has to implement technologies to the standards, those are most of the time commercial interests.
Apache vs nginx, rhel vs debian, spring vs j2ee, etc. etc.
I think it's good for the young crowd to get into self-hosting. You learn more fundamentals that way, for sure. But I'm not looking to move stuff into self-hosted.
I get unlimited scalability in AWS. I can standup and run an app for $0-$5/month depending on how I want to build it. That app will have unlimited scalability, redundancy, automated backups, etc. It's not worth dealing with the headaches of managing things myself.
Most everything is the way it is today for good reason, it's just that the youngsters haven't had to learn those reasons themselves.
We don't need to fix hosting. There's no such thing as "portable" or "generic" and there never will be. At a certain point you have to use an implementation of and that's going to lock you in.
I hate when people put things in quotes as if they're quoting something but it's not even a quote of anything.
Maybe go read the user policy instead of the headlines and the snippets. If your attention span can't manage that, get off the computer a little bit.
reply