Yeah. Basically IMO/IME the current best practice is to start the LLMs out with minimal/no skills/instructions/docs.
Notice the things they struggle with and the small problems (typically, environment issues IME) they repeatedly encounter and re-solve across multiple sessions. That is what your instructions should cover. When possible, move those instructions into skills, so they get loaded into context selectively instead of on every session. (Example: instructions for running specs, placed into a skill that only gets loaded into context when it’s time to run specs)
Again, this can be automated by the LLMs themselves: both Codex and Claude (and I’m assuming other major harnesses) know how to read their own transcripts and are good at looking for repeated friction and making concrete suggestions to reduce that friction in the future.
It takes a bit of a time investment on the user’s part, and every now and then you probably should throw it all out and start fresh so that the new batch of instructions can be appropriate for the current state of the repo and the capabilities of whatever model(s) you’re using.
Honestly, don't agree. I think even this is a waste of time and context.
The LLMs will always have these suggestions on things that will "reduce friction in the future" but I really think the LLM is a sycophant glazing you. Because even when you add those skills or prompts at some point the LLM ignores it or does whatever it wants to do anyways. Better to leave most of that out and deal with it when it arises.
Even your example, instructions for running specs... none of the current frontier models need this at all.
I think they're using Skynet. They seem to be claiming that part of the reason for not giving the LLM directions is because the LLM won't follow them anyway.
That must be a remarkable model. Smart enough to flawlessly discover everything on its own in every session... but also it just straight up just doesn't obey direction.
I'm saying a lot of devs/engineers little rain dances aren't really bringing the rain.
Keep a basic CLAUDE.md/AGENTS.md that keeps high level details. Have a way to reference other projects/apps/codebases to use as reference architecture.
Don't keep tons of markdown files. Don't keep "decision" documents (these are probably the worst). Don't make skills for every little thing (superpowers are dead these days, stop using them). Oh yeah, don't have the LLM generate docs for reference later... if it can generate the docs it doesn't really need them.
Nope. Opus and Sol for the most part. Haven't even used Astra or Fable.
At work it's all Anthropic. I spent a while on Sonnet models but have been pretty consistent about using Opus now. For personal projects I bounce between Sol/Terra and Gemini Flash.
Most of the models are good enough now. You don't need all these documents, memory, plugins, skills, etc. MCPs are good for hooking up to external sources of data (at work we have gitlab mcp, datadog mcp, and our knowledge base mcp... which the knowledge base is mostly worthless now because of all this AI generated documentation, but I digress). I DO still use beads on a lot of projects, but not always. My prompts these days are lazy as fuck: "review this repo, check out this other repo with architectural patterns we should follow and libraries/terraform/etc we should use, . What do you think? Let's review everything and discuss before we start building"
Oh no, I might have to stop the llm agent on occasion, or review what they did and tell them to fix a couple things. Way better use of tokens and my time than creating some rube goldberg machine. Way better than keeping a giant decision file that goes stale and causes context corruption because the llm only saw the OLD decision and not the UPDATED decision.
I understand what you're railing against in general: engineers who make these big memory/doc systems that quickly go stale and are at best useless and at worst actively harmful. Even more insidiously, these engineers are blind to this fact because maybe some of this stuff was effective 6-12 months ago and they haven't reassessed their workflows. Now that's dogma. We really do agree on that.
However. At least in the projects I've worked on, it's trivial to read the transcripts and notice that without enough direction, there are certain things the agents will burn lots of tokens to rediscover in every session. That's absolutely not dogma.
I think there's a balance between the tokens burned up rediscovering things vs tokens being used keeping docs/decisions/skills/etc.
And I think a lot of places are burning way more on the latter than the former. I think the former is more cost effective due to caching as well. So I HEAVILY lean towards the former especially with how effective I think it is these days.
I think there's a balance between the tokens
burned up rediscovering things vs tokens being
used keeping docs/decisions/skills/etc.
This mystical balance isn't difficult to measure. You don't have to believe anybody; just read the darn session transcripts. It's going to take at least one (but usually multiple) turns of grepping or other exploration to rediscover a thing. How many tokens does this add to your context? Since 1 token = ~4 characters you can count characters in your text editor.
Is this less, or greater than the number of tokens it would cost to simply add that thing to AGENTS.md?
If you're burning 5000 tokens to rediscover the same thing in every session, and it would take only 500 tokens to just put the thing in your AGENTS.md, it's an easy decision. Do you understand that paying 500 tokens in every session is less than paying 5000 tokens in every session?
If you're burning 5000 tokens to rediscover the same thing in 50% of your sessions, it's probably still worth it, but now you might consider placing it into a skill so that it only gets loaded into the 50% of sessions that need it.
If you're burning 5000 tokens to rediscover the same thing in 1% of your sessions then yeah, definitely skill. (You can add it to your prompts manually too, of course, but that is functionally equivalent to adding it to a skill that is set to manual invocation only...)
So I HEAVILY lean towards the former especially with
how effective I think it is these days.
How is this different from what I already said at the beginning of this conversation, where I recommended beginning with minimal (or no) skills or instructions... and then judiciously adding very lean ones based on actual observations? You claimed that this was a waste of time.
See here's the thing, you're pulling these numbers out of your ass when I'm making these recommendations from my observations that it's basically a wash and you get better results just having the agent read the code.
You aren't putting into 500 tokens what takes 5k tokens to discover. I've found that when I had these larger markdown files with all the details the models STILL went to the code to review. Then the markdown files get stale in spots, NO MATTER WHAT. Not only stale, but if you're having agents maintain these files they WILL hallucinate or input superfluous or unnecessary info. Now that I have short markdown files (or none in some instances) the agent just looks at the code and it's pretty simple to figure out.
And yes, I think most of the skills are a waste of time. You're just creating more shit to maintain. The most usable and reusable skills end up being short and concise, to where you could just type them into the prompt in any number of ways adhoc... effectively making the skill mostly worthless. The big skills end up being too much and less effective ime. And it can be difficult to rely on agents to invoke skills or the right skills.
If my skills need maintenance and hand holding, what am I really gaining?
As I optimistically hope you may be able to eventually understand with what I assume to be significant effort on your part[1], the numbers are going to be different on a case by case basis. Thus, my generalization.
This conversation is over, as you have been moving goalposts in various random directions: careening between "no directions for LLMs!" and then talking about your "minimal" AGENTS.md and markdown files, and then disagreeing with me for having minimal AGENTS.md and markdown files, in favor of your approach which involves... minimal AGENTS.md and markdown files, apparently. Luckily, it all sounds so schizo that at least nobody will take you seriously enough to follow your bad advice. It's exhausting talking to you, but surely not as exhausting as being you, so I do still have sympathy. Good luck.
---
[1] remember to hydrate, a good night's sleep helps too
Notice the things they struggle with and the small problems (typically, environment issues IME) they repeatedly encounter and re-solve across multiple sessions. That is what your instructions should cover. When possible, move those instructions into skills, so they get loaded into context selectively instead of on every session. (Example: instructions for running specs, placed into a skill that only gets loaded into context when it’s time to run specs)
Again, this can be automated by the LLMs themselves: both Codex and Claude (and I’m assuming other major harnesses) know how to read their own transcripts and are good at looking for repeated friction and making concrete suggestions to reduce that friction in the future.
It takes a bit of a time investment on the user’s part, and every now and then you probably should throw it all out and start fresh so that the new batch of instructions can be appropriate for the current state of the repo and the capabilities of whatever model(s) you’re using.