Let's not forget the Bitter Lesson. Small models sound really nice but at some point you're just fighting the laws of information theory. Efficiency gains on the small model side are nice, but efficiency gains + giant model tends to be even better...
More compute/larger datasets during training != larger models. The Bitter Lesson was that just scaling things up beats custom hand-crafted optimizations. Up until 2024, we thought that meant scaling up the parameter count, but then that started to plateau. After o1 was released, we thought it was about scaling up test-time compute. Now, seeing how 30b models can easily outperform 200b models from a few years ago, it seems like what we need to scale is RL, at least for agentic capabilities. It looks like 30b is already enough for good agentic capabilities. Larger models aren't considerably "smarter" (especially since they're mostly MoE anyway, with something like the same 30-50b active parameter range), they just know more (better world knowledge), which lets them make more informed decisions. Maybe we just need to scale up the retrieval layer.
I'm just saying that any larger model with all of our optimizations of today will always beat a smaller model with the same optimizations. Until the smaller models + optimizations are at AGI levels I don't think anyone will really care about what smaller models can do other than for routing / toolcalling
I feel like the industry has quietly moved past the Bitter Lesson. In 2023 the story was naive parameter/data scaling and “emergent” intelligence properties. But there wasn’t enough data or compute to keep pushing in that direction, and the gains from it have been sublinear anyway.
Now, the labs spend enormous effort curating data pipelines to fit the models to a large assortment of very specific tools, tasks, harnesses, domains, etc. They also kind of fit to benchmarks by creating loads of synthetic training data that resembles benchmark tasks. None of this feels like the “scale up primitive methods and turn off your brain” message Sutton originally delivered.
You're totally right that we're far away from brain-level efficiency, but I'm just saying any efficiency gains we make towards small models will likely be felt on large ones as well, and we'll all move the goalposts to what the frontier can do. Hypothetically, getting GPT 5.6 performance in a 30B model would be amazing, but just imagine what you could do with a GPT 5.6-sized model at that point.
