> We have decided to organize these like departments similar to the way you might hire out humans. I'm not 100% sure if that's the best approach, but I will say it's been easier for people to understand because they're more mentally easily able to traverse the bot org chart if it somewhat reflects a traditional business org chart
I think this way of thinking is going to be very important to drive adoption of AI systems because the human analogies benefit from the pre-existing domain knowledge and expectations of people.
I like using the exam analogy for evals as a qualifier for work for your "AI hires" so you can trust them to work on a specific domain.
I'd be quite curious to see what your approach to evals/testing/tracing and agent/system mutation is.
- Keep them organised in software repos that you install with symlinks for all coding harnesses that you have. Progressive disclosure based on the frontmatter does the rest.
- I make sure they work with AI evals. Think of them like integration tests to prove behaviour. They're useful to optimize your flows. I try to make my skills be mostly a translation between natural language and good small fast tools that they call.
- I change them as a new problem arises. Not just because.
Skills can't be eaten by model capabilities if skills represent a workflow that is custom to my team or my person.
People always say this about the evals, but I find it hard to have a practical implementation of such a thing where you won’t end up spending 100x the amount of time on the evals than building the skill itself.
Like, ok, I have a debugging skill, now how do I make evals except for the most trivial things?
You don't. If you're using skills to force the AI to fullfill some must criterias, it's not going to work. Must criterias need deterministic checks -> be it hooks or what not.
This is also my biggest gripe with AI. I.e. for specifications, no matter what hype machine I tried, it never fulfilled my criterias, which are: easily verifiable, concise, small specs. Hence I built https://github.com/RicardoMonteiroSimoes/Yamlet initially for claude code, but then decided to use extend it for pi.dev. I now have a dedicated docker image for pi.dev, that only contains Yamlet plugin, and whenever I work on spec I spin it up.
The end result is a .yaml file that easily works in git + git diff, so that I can then proceed with the technical specs-
Why do you have a debugging skill? Just tell it to read the docs.
Skills are for packaging instructions for how to interact with your organizations homebrew process and tools. By definition skills shouldn’t be useful outside of your org because they’re just docs and third party tools already have them for humans.
Rather than teach the agent where grafana is, how to use a sentry trace id to find a otel traceparent, where my ALB is, I'm what clusters do what, which namespace prod is in, what our stack looks like, that thing that looks broken actually isn't, etc etc etc. I just paste in the sentry trace id, a guess of what might be wrong ("i think we overtuned gunicorn again" or "developer bob pushed short sha 123456 and nothing is working") and say "use your investigate skill" then get up and go get a coffee and usually by the time i get back i have an investigate doc filled out from a common template in my notes repo. The agent almost never spends any time spinning it's wheels finding out what the various environments do, where they're located, what our metrics and logging looks like, how to access it etc etc.
To be fair it's the only skill i have/use but I got real tired of explaining the same 12 things over and over. Having it document every incident means I have a dense library of every problem we've run into over the last six months which helps identify recurring problems for RCA
There are also skills that help LLM do the thing it can do without the skill, but faster (by cutting out unnecessary discovery). I guess for such skills the fail case is "being slow"?
I imagine many fail cases can burn a lot of tokens/usage/time because failing LLMs can be very persistent. Maybe some upper bound (turn count, timeout) would help too.
I am starting to wonder if I am doing something wrong: I ignore evals and instead I just try new models or new harnesses (or tweak my own harnesses) by solving problems I want to solve in any case; I just use new tools and form my own subjective opinions of them.
When I say evals I mean the evals you write that verify that your use cases are upheld. Think of it like a regression test for different behaviours/user stories.
The idea would be that if you already know what you want from an autonomous system, you don't need to verify manually every time and instead just run these tests to see if there's any regression of any kind. Generally I recommend structure output and evals that are just a plain assertion, if possible. Cheaper, faster, deterministic assertions.
Big yes on this. I do not understand the appeal of skill shopping. The one exception I have is things like the Axiom Apple development skills and e.g. the official Flutter skills. At that point the skills are just docs though. It's either I remember to paste a URL to the official docs or I just install the skill. But shopping around for random skills just sounds extremely unappealing.
You can also use PI with local models like Qwen3.5-35B-A3B [1] and they can be surprisingly good if you development in minimalistic/simple ways.
One game changer when it comes to tweaking configs that are optimized for your use case is that you can easily use a "more powerful" cloud model to identify a good enough config for your local server/pi settings combination [2] in a pattern that applies pretty much anywhere.
Skills represent a pattern many of us converged to before it was named as such (very useful for comms): Progressive disclosure, determistic scripts bundled (not just markdowns). See past comment:
This is simply a different kind of AI than LLMs will ever be. There may be some kind of architecture that does this in the future, but it's not, and can never be a neural network that is attempting to be AGI.
Raw materials in, lots of process steps in the middle with tolerances etc; some % yield of goods out.
When you work with LLMs, the "raw materials" is its first attempt to answer your prompt.
Mass manufacturing mirrors what we do with LLMs to knock their output into shape: grounding, automated quality checking tooling (eg. linters for code, grammar checkers for prose), and ultimately, as many turns as needdd to manually bang the output or work product into shape...
You should never ship your AI's first draft - that'd be slop. But you also shouldn't have to repeat yourself to get its output into a form you can use. If you know you're always going to have to tell it to change one specific thing, see if you can automate a process that tells it to change that thing for you, so you don't have to. Do this enough times and you've created a deterministic outcome at least at some level. And you're out of that loop of getting it to that level.
Tooling for this is limited today. You can provide instructions; but you're always at the mercy of labs to make models that follow instructions and at their mercy that they didn't put conflicting instructions in the system prompt.
Better to put in things like linters and grammar checkers - even going as far as putting in a fact checking process or for legal stuff, a citator.
Without these extras, AI output is no more real than some dream you had.
I could but I'd probably miss the angle that is helpful to you. Do you want to instead share what your pain points are, what are you trying to delegate to AI and what you aren't? Based on that I can expand my answer better with a thing or two you can try.
This thing will be far more specific to whatever usecase your thinking of. It wont generalize as much and as easily as LLMs do. Youre heading back towards IBM's watson
Many of us were saying it a year ago but now with model restrictions (e.g. fable) and pricing changes it should be obvious to people that part of the economics is avoiding vendor lock-in with evals (so you can easily switch providers/models/optimize) and increasing control by investing in local models which could be good enough for your tasks, at whatever the price for your cloud compute is. Eventually consumer hardware will also be able to run good enough.
You can use Big/Cloud LLMs to help you "find good enough configs" for your local/small llms [1] and stay quite nimble in the face of rapid change.
Every Linux user or FOSS enthusiast knows the acronym FUD: Fear, Uncertainty and Doubt, which were a set of techniques commonly used to disparage efforts of open source communities. Linux was evil and anticapitalist and we needed to use "CorporateTool" and ban/restrict Linux.
The same companies later would be running their entire infrastructures on it and on open source.
With AI, open weights and local models, we will see the same claims, even if the named fears change.
The end users and humanity are better served by collaboration and openness than by creating oligarchies.
Some tools used to be 0.x for ages and, in this case, the 1.0 signals they're happy enough and allows them to promote things in a better way.
This (edit the durable part) is I guess the natural evolution of playing around building temporal like things for a need that many have.