To be fair there's probably a considerable amount of engineering that went into evaluating those markdown files so the agent behaviour is statistically reliable. The markdown is the product, not the process
I do, I like to know how badly my agents' context is wasted and what unexpected side effects to watch for (like, "always start with ${clitool} --help" == always waste few hundred tokens when even touching the skill; or instructions asking it to do something that generalize into stupid thing in larger context).
If those skills were unreadable, however, that would imply proper engineering - like e.g. the skills themselves being an output of iterative RL over set of evals.
I don't think unreadable skills implies proper engineering at all. It's just as or more likely that they're the result of a blind iterative process with no clear improvement signal. (And whether iterative RL over a set of evals is actually proper engineering here is another question...)
> And whether iterative RL over a set of evals is actually proper engineering here is another question...
I'd put it like this: regardless of the merit of how they're applied, it would at least demonstrate possession of the advanced skills expected of experienced software engineers.
I had more success starting from scratch. Especially claude's skill-creator skill, it micromanages, which is in fact worse for 5+ models than just leaving the instructions out and crossing your fingers.
Start from scratch, do some test runs, find the bugs, add the minimal possible text to avoid the bug, iterate
You can get 95% of my impl workflow skill by just telling Claude "split the work into slices and use ephemeral subagents" and the other 5% takes like 10x as much text to achieve
Markdown can never guarantee deterministic agent operations. It is an influence on inference, not a deterministic code path. How "statistically reliable" is it?
Write a prompt, evaluate the prompt, understand that is succeeds 95% of the time.
Write a new prompt, evaluate, it now succeeds 99% of the time. Measure what changes between prompt #1 and prompt #2, understand what contributed to the performance jump.
Write a third prompt, this one succeeds 100% of the time. Increase the size of your evaluation set, find a 1/5000 error-class and a 1/10000 error-class, add some explicit code to correct for this cases.
Roll out to production, collecting usage metrics. You make some tweaks to your harness, your prompts. Eventually you have confidence that your system has fewer mistakes than 1 in 100k.
Now, multiply this iteration across all your different prompts and different ways that they might interact with one another.
There's a reason engineers are prissy about people coming along and saying "I write code, I'm an engineer" that people periodically try to sand-paper away.
Engineers don't just tie a sheet to a rock and throw it off a cliff and call themselves aerospace engineers.
They do full diligence on the theory, math, physics, material science, fluid dynamics, etc, and plan a controlled series of tests specifically designed to verify/challenge/disprove their concept and the theories behind it.
Sure, there's a team member ultimately responsible throwing half a dozen rocks off a cliff in the first test.
A technician.
The guy who throws the rock off the cliff is a technician.
The other glossed over part is that the above sounds like science.
Engineering often continues until the concepts and theories are developed into safe, practical methods. "If you stay within these parameters, you can confidently expect these results." The reliability can be codified and reproduced without going from first principles on every application of it.
It's not clear to me that the current AI fad is really developing such reproducible, safe methods. "If you stay within these parameters, you might get these results. Or a teapot. Or some subtly misleading fabrication."
You have to do full due diligence to validate every result. There is safe usage where the hard work was done up front so that day to day practice can skip to boring and reliable application.
To a software engineer, a (current) LLM is a stateless algorithm that performs an idempotent transformation on a large numeric input.
People who think it's a system that thinks and reasons have confused the agentic harness, perhaps forgotten(?) layer0[0] is a seed, the inference engine sets to a concrete value when the caller leaves it as 0.
They probably work on (current) AI software by repeatedly writing prompts like "DON'T READ THE FILES IN /tmp. SOME OF THE FILES IN /tmp ARE VERY LARGE. DUE TO THEIR SIZE, YOU ARE NOT TO READ THE FILES IN /tmp." and wondering why the model becomes obsessed with files in /tmp 100k tokens into every conversation.
"Most classical engineering fields deal with probabilistic system components all of the time. In fact I'd go as far as to say that inability to deal with probabilistic components is disqualifying from many engineering endeavors."
This is cope and fundamentally misrepresents engineering. Engineers deal with a problem space that is probabilistic (although they try to model it as best as they can), but design solutions in a deterministic space. With LLMs the solution space itself is non-deterministic.
> It's not engineering if you're just guessing as to what is degrading the performance and what might improve it.
Engineering is literally the art of making educated guesses and then testing/proving/disproving/improving upon them. Nothing is exact. Everything is approximate. Iterate until the result is good enough.
This is false. A bridge is not built with approximations, it is built with a deep understanding of structural physics. Yes there are some unknowns, no it's not "educated guesses".
If you can identify gradient (what direction your change will impact the ultimate goal), then just repeating the process (or reverse-process) can find local maximum.
Still it can be a software engineering if the gradient candidate / measuring gradient / repeat process can be done at scale.
Things like Voice to Text and biometric unlocks (fingerprint scanners, face ID) have worse success rates and they're used every day by billions of people.
The fundamental issue is, that "we" somehow decided it would be a good idea to throw all the fundamental ideas of computing (determinism, context, separation between data and execution,...) away and try to solve the issues by running a probabilistic/stochastic word generator on top of deterministic circuits instead at 10 magnitude worse efficiency.
It's not a fundamental issue. Determinism and "separation between data and execution" are artificial constructs, make-believe universe in which we design classical code, and a whole lot of hardware engineering goes into allowing us to briefly forget it's all fake.
Real world is probabilistic in practical / metrological, if not fundamental sense, and separation between data and execution does not exist. Our reality does not support such separation.
> a probabilistic/stochastic word generator on top of deterministic circuits instead at 10 magnitude worse efficiency
It's 10 magnitude better efficiency end-to-end, if you factor in design time you'd have to spend to get your "deterministic circuits" (which really aren't, we just paper over it) into shape so they deterministically solve a specific problem, for each problem you want to solve - where with the "stochastic word generator", you just need to change the text prompt.
It might not be apparent from the start what are the best demands to put inside a skill, you can only know by evals. There are whole papers dedicated to changing a few details in a coding harness. https://arxiv.org/abs/2609.20519
Engineering is the use of mathematics to turn science into technology. Statistics is mathematics, comp sci is science, and technology is the end product.
Citation needed. Have you read some of the skills slop Anthropic were pushing at some point? Here is "frontend design":
> Consider Chanel's advice: before leaving the house, take a look in the mirror and remove one accessory. Human creatives have memory and always try to do something new, so if you have a space to quickly jot down notes about what you've tried, it can help you in future passes.
How about "canvas design"?
> THE ESSENTIAL PRINCIPLE: The topic is a subtle, niche reference embedded within the art itself - not always literal, always sophisticated. Someone familiar with the subject should feel it intuitively, while others simply experience a masterful abstract composition. The design philosophy provides the aesthetic language. The deduced topic provides the soul - the quiet conceptual DNA woven invisibly into form, color, and composition.
