Except compute

Do you think we are near the theoretical maximums of how much model we cram into 20B?

I don't consider myself qualified to make that call, though I believe there's plenty of room for improvement. But I don't mean inference compute, which is fairly accessible for well-off enthusiasts even now. Training is a lot more expensive though.

Back in the day we had seti@home, prime95 and fold@home. We will make it work.

When models are in the hundreds of gigabyte range, there are not enough nodes to train, because most people dont have 100k in AI hardware. Also there is no training method for machines scattered around the world as far as I know.