It's funny you mention this, because I've been thinking lately that all the focus on code quality benchmarks for LLMs seems to miss something. In my experience, LLMs aren't actually all that good at prompting other LLMs, and that's exactly the skill that matters for proper subagent usage.
The corpus of code available to train LLMs on is rich in both volume and quality. The same can't be said for LLM prompts. Prompts only came into existence alongside LLMs themselves, so most of the prompts out there are actually LLM generated, and training on them causes every model to overfit to the same patterns.
LLMs can give clear and concise directives to other LLMs, but only for basic tasks. Beyond that, they frequently overstuff their prompts with too much prose and too much direction, and worse, they often fill them with subtle contradictions.
Of course, there's a spectrum here. Some models turn out to be better at this than others, and I suspect this is what differentiates Fable-type models from the rest, and why vendors have started keeping their models' reasoning confidential.
Quick tip here: Where possible try to trick the model into thinking whatever task it is writing will / could be read and performed by one of your coworkers rather than another agent. It will quickly fall into line and stop writing prompts meant for dumb historic low intelligence LLMs.
The "agent factories" available to employees at the labs and hyperscalers clearly prove that agent delegation does work. Today, maybe that capacity lies with Fable class models, as you pointed out, but there is no doubt with opus 5.5 that this capability will be available to cheaper models as they get better.
I think this is one of those areas where the translation from research to the public ends up too precise on the details, beyond the cognitive load most people are willing to commit to it day to day.
In the same spirit of upāya from Buddhism, I like to compress this type of topic into a simpler model, even if imperfect.
Here's how I model cancer, and how I judge whether something is likely to give me cancer, without trying (and failing) to memorize a million risk factors individually:
You are a collective of cooperating cells. When cells rebel, the order that holds your body together collapses, and you get cancer. Cells generally don't want to rebel. They're content as long as they get their energy, their paycheck.
But what about activities that cause frequent cell death? A sunburn, smoke in your lungs, alcohol in your bloodstream, radiation (essentially atomic shrapnel flying through your tissue). Anything that causes inflammation, especially chronically. When cells die before their expiration date, the survivors have to keep dividing to replace them, and they start to get pretty mad about it. Maybe even rebellious.
With every round of division, they are in essence "evolving" away from their cooperative nature to cope with a hostile environment. A cell doesn't care whether the collective lives or dies, and it doesn't have the capacity to reason about that in the first place.
So the question becomes: are you keeping your cells safe? Are your activities hurting them? If they're not happy, they'll rebel, and you'll die.
This is where the model earns its keep. Take obesity. It's on every risk list, but excess fat isn't what makes cells rebel. The real culprits are the conditions obesity creates: chronic inflammation, more cells that need to be maintained and replaced, a body that is generally harder to live in.
Weight loss is on the list because it's the easiest way to fix those conditions, not because thinness is protective in itself. Once you see that, you don't need the list. Anything that makes your cells' lives harder, day after day, is raising your odds, whether or not anyone has published a paper on it yet.
The math is real, but a model like this tells you which variables you actually control.
Somewhat tangential, but this reminds me of when I took a certain Microbiology midterm in college. I hadn't put much effort into that class leading up to it, as I had some tough CS classes that consumed my focus that semester (also was admittedly not a great student in general). So when I finally went to take the midterm, after cramming like mad before the exam, I quickly realized I basically didn't know the answer to most of the questions, even what I crammed was only a small portion of the material.
The exam was huge, at least over 10 pages, and even when we technically ran out of time, the professor was kind enough to move the remaining exam takers to the neighboring lecture hall to continue taking it. I recall I spent a total of 2 hours on that exam.
Now mind you it was mostly short answer or multiple choice questions. The multiple choice questions were pretty sharp too, lots of traps and false but sounds right answers mixed in. But if it had been purely essay questions, I would have been screwed.
However with such a huge corpus of information in front of me, I ended up basically learning all the material on the spot. I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
In the end I got B+ in the exam (which was curved to an A), and walked out understanding the material better than I did walking in.
Reflecting in the experience years later, I've wondered if a hypothetical LLM which was ignorant of microbiology could do the same thing if fed that exam. In some respects the traps they placed in the multiple choice questions actually were what helped me refine my understanding the most. Made me appreciate information theory more.
> I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
This strategy saved my ass on so many tests from high school through college that I did not study properly for.
I think it would certainly be possible to write a test such that this strategy doesn’t work at all, but lucky for me no one did in any meaningful way.
There was a couple famous papers from a few years ago asking why computer science students, sorted by ability, seemed to form two clusters.
In one of the studies they got incoming CS students to fill out a multiple choice exam on programming, before the class had even started. Then they looked for hints in their exam results to see if they predicted the end of semester grade.
Turned out following this strategy was an incredibly strong predictor of success in computer science. Some students somehow got 0 (or close to it) in the multiple choice test. But the researchers looked at their answers, and could see that the student was forming hypotheses about the questions. Even if they got all the answers wrong, they answered all of them with a consistent idea of what the computer programs might have meant.
Weak students just answered each question individually based on vibe, without using what they learned (or figured out) from other questions.
> I just kept doing multiple passes through it, each time I noticed one of my answers contradicted one of the others, I would make adjustments to harmonize, which indirectly refined my understanding.
I used to do something somewhat related during exams: I could never find contradictions/mistakes while working forwards, because somehow my brain would make the same mistakes consistently every time I go through something in the same direction, no matter how slowly and methodically I would do it.
So I worked as fast as possible in the forward pass and then reversed the problem in my head and worked backwards just as fast. Except this time I would often find the expected inputs would no longer match the actual inputs, so it helped me identify a lot of mistakes.
Yeah I walked into this with my pitchfork ready but isn't this stuff pretty... reasonable?
McDonalds is just a restaurant, they predict what menu items you like and how likely you will come back, is that really a problem? What is the concern here? I don't use their app but presumably for those that do, isn't this exactly the type of data that would simply influence whether the app offers a coupon or something? Or how well a franchise is doing? Maybe also what to keep on the menu or how much to stock?
What exactly is the bar here I wonder? McDonalds has security cameras, it also has menu kiosks with cameras (I think?). The alternate reality evil tech-like McDonalds could have theoretically used facial recognition to categorize all footage of any given customer, then used this footage to estimate their friends, spouse, BMI, age, gender, ethnic group, attractivness, disability, etc. They could then sell it to third party data brokers for extra cash.
That would certainly be the kind of dystopian surveillance to get me swinging my pitchfork around, but this? Having assumed the worst I'm now walking away realizing McDonalds might not be so bad after all.
The corpus of code available to train LLMs on is rich in both volume and quality. The same can't be said for LLM prompts. Prompts only came into existence alongside LLMs themselves, so most of the prompts out there are actually LLM generated, and training on them causes every model to overfit to the same patterns.
LLMs can give clear and concise directives to other LLMs, but only for basic tasks. Beyond that, they frequently overstuff their prompts with too much prose and too much direction, and worse, they often fill them with subtle contradictions.
Of course, there's a spectrum here. Some models turn out to be better at this than others, and I suspect this is what differentiates Fable-type models from the rest, and why vendors have started keeping their models' reasoning confidential.
reply