"one-shotting" is fun for a low (human) effort toy, but it's a mistake to get overly hung up on LLMs getting everything right on the first try, since a lot of that is the harness and the internal prompting that makes it burn extra tokens to test everything to get it in that one "shot" (ui turn).
While it's nice that LLMs are getting better at arriving at the desired deliverable with fewer surprises (or broken things) doing it is not necessarily a sign that the model is actually smarter, just an indication that what you are doing is likely a well-trodden path for the model, the RL and the harness.
I have a personal benchmark suite that runs a bunch of difficult algorithmic problems in 3 tiers: vibe coded prompt (just a few sentences), vibe coded prompt plus a description on what the solution will be evaluated on, and finally a fully specc'd prompt that leaves little space for interpretation.
The current Fable models are approaching a level where the -eval runs perform systemicallty better than the "clear spec" runs. (And right now its just Fable 5.1 and Opus 5.5 that behave so well, all the other models I tested - all the OpenAI and the leading Chinese models - work better with clear specs). I'd encourage everyone to experiment a bit; I am running my serious workflows with labourious force-me-through-all-the-design-questions mode, whereas my hobby projects, some of them hard, I am trying to see how much I can get away with with vibe-grade prompts. My conclusion is, with Fable its starting to look like what Vibe coding was promissed to be a year ago - a magical box that generally does things very well.
Btw, congrats to the OP -- the game is enjoyable and I played a few levels. I dont remember when I last played any games for more than 5 minutes. Good stuff.
My experience has been that cruft accumulates and creates back pressure via growing token usage.
The APIs aren’t clean, which creates indirections to other parts of the codebase to check on leaking abstractions. It doesn’t learn, though, so you pay for those indirections every time (right now, anyways).
I’ve not been impressed with its ability to clean that up with a generic “make it gud” kind of prompt. I have to point it at something crufty, but it will fix it then
I agree, but for some projects, that first vibed version is just about good enough and becomes the final version. I have a few things like that and it was nice not to have to bother.
For work stuff, I don't dare 'vibing' anything, and force myself through torturous design cycles and regular refactoring runs. But thats serious stuff with hundreds of thousands of users.
Btw. Fable 5.2 seems to burn through tokens with gusto. I ran my benchmark suite on it, added a complex feature to a hobby project (a gothic cathedral designer), and voila, 2 resets gone.
I think one-shorting avoids some of it because most of the project should still be in context.
I’ve noticed because I’ve been doing Clojure and turns are taking notably longer due to unit tests because it doesn’t really grok its own system anymore. It is like 20k LoC at this point with unit tests lol
For nearly all projects a trash version is good enough for V1. That has been true long before LLM coding. I think that fine as well. My home server doesn't need a monthly tested disaster recovery plan
I've been searching for the right term for this; you've nailed it: back pressure.
Whether the "cruft" is test accumulation, multiple conflicting architecture approaches, a bazillion ADRs/"laws", or just pure code sprawl, the outcome is token and time waste which gets worse as it compounds. And it compounds quickly.
I've tried automated code review through multiple lenses, more ADRs, instructions out the wazoo, but at the end of the day I just cannot figure out how to replicate the "through-line" consistency that responsible humans who need to maintain a system generally create.
Of course, most software projects do eventually fall victim to this same problem and need to be rewritten, but it takes place on the order of years, and by the time you need a rewrite the business has changed enough to demand one anyway. But with AI-owned products, this can happen in days or months.
I’ve been going sort of hybrid and using more targeted prompts. I skeleton the API most of the way and have the LLM fill it in and implement. This is at the code level, not the HTTP API.
Most of the time, anyways.
It keeps me in the code enough to feel the friction of bad APIs but high enough that I’m not reviewing every line of code.
I also try to scope my reviews to only the intersections between systems (SQL schema, library APIs, CLI flags, etc).
I’ve got nothing on the tests. Mine writes a million of them. Some are brilliant, some are stupid.
Like it once did TDD by storing transformations to the same base object in a list rather than specifying the whole object. Clever.
The very next test was an arity test to make sure it threw errors if you gave the function the wrong number of arguments. In a webapp, not a compiler or something. Stupid. Do not unit test the language’s runtime lol
The process of using AI to create things changes how you evaluate the quality of what's generated, so really the answer is "until you accept the output first time."
That's a combination of AI improving and your standards changing.
They also change because of (1) how cheap things have become and (2) how easy it is to iterate. If I paid $500k for an app from a good dev team, I will worry that anything I don't declare a bug won't ever get fixed, or will cost me dearly. If I get the same app for $300 in tokens and 2 days of someone's work, I am fine with what I am not sure about. It will cost me next to nothing to keep iterating it until I am satisfied.
That’s astute. I’ve increasingly felt over the last few months that people just don’t particularly value quality. The bar for most people is much lower than professionals (in whatever domain).
I definitely remember a game I played for more than 5 minutes recently. A lot of those were not AAA(A) games but rather indie or previously-indie-now-got-a-publisher-to-notice games. None were "AI one shots" that just built something that looks like a re-skinned version of "Teenage Mutant Ninja Turtles: Shredder’s Revenge", which I'm sure is pretty abundant in training data? And that was a great game btw!
> "one-shotting" is fun for a low (human) effort toy, but it's a mistake to get overly hung up on LLMs getting everything right on the first try, since a lot of that is the harness and the internal prompting that makes it burn extra tokens to test everything to get it in that one "shot" (ui turn).
I'd be more specific and state that these models are now way far from being LLMs and instead they are full blown agent orchestration systems capable of running multi-layer ReAct loops and tool-calling, and with your whole session history as context. It doesn't make any sense to refer to them as glorified chatbots as the LLM part is nowadays far from being the component that delivers the highest value.
> as the LLM part is nowadays far from being the component that delivers the highest value
Yes it is. The rest of the agent could basically be a while loop, a two sentence system prompt and a shell invocation, and it would still work reasonably well.
These agent swarms with harnesses are certainly more than just an LLM, but the LLM is certainly the integral part that delivers the most value. Without it, there is no AI.
With a LLM alone, all you have is a stateless chatbot. If you increase the parameters of the LLM by a few orders of magnitude, you still have a stateless chatbot.
If you start with an anaemic LLM and run it with a ReAct loop and tool-calling, you now have something you can use to do agentic coding and RAG-like capabilities.
There's a reason why relatively small open weights models are starting to perform on par and even outperform models that need a huge datacenter to run.
Which ones, and which models? https://openrouter.ai/benchmarks/gpqa-diamond shows Kimi K3.1 (not small) roughly on par with Gemini 3.7 Flash. I see no coding benchmark there that could substantiate your claim, and it doesn’t match my experience either, so I’m curious what you are basing it on, but you’re leaving the onus on me to prove you right.
But one shot prompting does not mean one shot testing. I see continually that Fable can design and test prototypes before committing to a design goal and testing multiple layers until the goal is reached. Many of the tests are code unit or integration tests that can be re-used without token spend.
The idea that a word is defined by the company it keeps was decades old. Mikolov’s insight was that you could learn it at scale with a very simple model, and that meaning would fall out as geometry. Obvious in hindsight, foundational in practice.
Not to downplay his accomplishments (I have sung my praises in another comment), but even vector models were already around and somewhat popular before. Much of it started in Information Retrieval with things like TF-IDF vectors and later PPMI matrices that were converted to small, dense matrices with SVD. Of course, later Yoshua Bengio et al. introduced neural language models for training embeddings.
There is even a paper by Levy & Goldberg from 2014 that shows that skip-gram with negative sampling implicitly factorizes a PPMI word-context matrix.
But Mikolov's formulation is very nice in that it is a simple linear model that allows for a very simple Hogwild optimization that worked very well/fast on multi-core CPUs of the time.
At first I hoped that fears about ICE abuses were overblown, but then I saw video after video of masked, badge-free agents manhandling and throwing grandmas and elderly/weak people to the ground and generally acting like thugs, stalking immigration courts for people trying to comply, doing nazi salutes, etc. It quickly became clear that ICE is largely made up of KKK or proud boy thugs who see it as a chance to brutalize a vulnerable population that they despise.
Now that this data has been leaked, identities can be correlated with facial recognition, gait analysis, and other biometrics to conclusively identify the agents who committed crimes. Much of the video footage shows agents flagrantly breaking the law, abusing power, violating the rights of citizens and noncitizens alike.
Now that his leak has occurred, criminal ICE agents will be brought to justice for many of the crimes committed already, and the data will also inform the prosecution of crimes they will likely commit during the rest of Trump's term in office. Many of the crimes are violations of state law, so there is nothing the executive branch can do to stop the prosecutions.
You can see fate of Guantamo's officers and their chain of command. As far as I remember, nobody was really punished, several people were dismissed, but with full pension.
Because it's not guaranteed, and anyone who shrugs and thinks, "Oh who cares? This will sort itself out. Why bother myself over it?" contributes to them not being held accountable for their crimes.
The backlash against agentic coding comes from the split that has already existed among professional software engineers. The ones who are most resistant to it are the ones for whom it has been "just a job", but the ones who love it are the ones who love solving problems. It's now possible for a small team (one person, a few) to solve truly daunting software problems with the help of agentic methods. If you were a 9 to 5er who valued craft but didn't dream about solving problems, AI is a buzz kill.
Im one of the more mid to resistant folks and this has never been just a job to me. I did take immense satisfaction from learning, deeply understanding how things work and the consist and repeatable behavior, which is just not there with ai. Especially the consist/repeatable behavior, which to me computers that behave inconsistently aren't nearly as worthwhile.
It's a shame you package your speculation inside an insult 'cause the "problem solver" as LLM booster isn't entirely wrong and aren't entirely bad.
A LLM seems like a powerful but dangerous tool. It's dangerous even in problem solving terms since one day can output something that advances your project significantly and the next day is can output vacuous verbiage that seems like a solution but is nothing.
They deserve a critical view but they make it hard to maintain that critical view.
> The ones who are most resistant to it are the ones for whom it has been "just a job", but the ones who love it are the ones who love solving problems.
Wow, what a simplistic, uncalled for view of the world. This is just so ignorantly and offensively incorrect.
There is nothing wrong with practicing craft, but when craft starts to matter more than vision/ambition (as it has for many engineers, even very good ones) there is going to be resistance to any method that de-emphasizes craft and emphasizes vision.
Even things like Ruby on Rails and node.js (in their early days) irritated some software crafspeople for the same reason, the tools felt like the happy path was somehow a mediocre one that was willfully ignorant of craft.
Suddenly, agentic coding has seemed to remove much of the craft from software engineering, and those who have been showing up every day celebrating craft are feeling it like a gut punch.
This is illusory bc craft still matters -- rust is a superb example of how craft matters more than ever in an agentic world, fwiw.
No, this isn’t what you said. You said that those resistant only view this as a job and don’t love solving problems - this is a completely ridiculous opinion/propaganda.
The world isn’t so black and white. It’s possible to be excited about what AI can help accelerate and simultaneously be resistant to the idea that 100% of software engineering (from premise -> shipped and maintained) can and should be offloaded purely to AI. And more importantly, AI reminds me a lot of leverage - it amplifies the good, but possibly more importantly it really amplifies the bad.
Software engineering is much more than just writing code.
I really don't think anybody in these "debates", on either side, has a high horse to ride with respect to rhetoric and solemnity.
This in particular is a really silly hill to die on, because literally every debate about software development, from AI to language choice to tabs vs. spaces, eventually settles into a gravity well of asserting that all the good developers agree with your take.
> This in particular is a really silly hill to die on,
Stating that people can't love software development or solving problems because they are resistant to AI taking over their entire job is a silly hill to die on?
I really enjoy what I do, and have done, for the past 20 years. When I first wrote code I literally thought I discovered magic. In many ways, AI has re-triggered some of those feelings!
I also really care about building quality software for our customers.
And it infuriates me to no end when I see huge amounts of objectively incorrect code being written by AI, and the developer behind it stating they have no idea what it's doing, or when people say they can't solve a problem because they are out of AI tokens, etc.
But supposedly these are the people who care about solving problems? Not just that - they are the only people who care about solving problems.
You can choose to take offense if you want, but since I've seen literally the same sentiment expressed about whether people choose Emacs or vi, I have a very hard time taking that offense seriously.
Oh yeah let’s play this game. The ones resistant care about their work and pour long hours getting the details just right. The ones who love it are the ones who just want to close tickets. It’s now possible for a small team (one person, a few) to create an avalanche of slop with the help of agentic methods. If you were a 9 to 5er who valued sprint velocity but never once pondered how a monad is like a burrito back in the day, AI is the All-Mother
In my opinion, you’ve got the situation backwards. The people who see it as “just a job” are the ones who like AI because they just want to get the work done and get paid. The people who genuinely enjoy solving problems, on the other hand, don’t like AI because it takes away the problem-solving part of the job.
reply