If small size, I recommend lots of plants and capping any "live" substrate with sand. Had one going with shrimps and a few endlers for years. I only topped the water up and very occasionally changed the water. Not that it really needed it.
After the first few weeks, it just soldiered on with minimal algae or intervention needed. It's a lovely thing to have, I can recommend it. (That said, I also have a small aquarium problem according to the wifey)
Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful.
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute
I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
(As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering").
TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this.
One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later.
There are a lot of other high level abstractions here to look at. Prime Intellect has one.
The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.
This guidebook covers pretraining, post training (SFT, RL) and a couple other topics. And others authors have also written books that fit on single node reasonable hardware.
If you want to start with a pretrained base I built Gemma 270m and released it last year. This fits on a raspberry pi.
The fundamentals of AI don't require industrial amounts of large scale. Think of it like this, when I was learning how a plane worked when I was a kid I didn't build a 747 at home, I started with scale sized model planes. Same idea here.
And FWIW I'm a staff researcher at Deepmind (opinions here are my own) so I want to specifically encourage all people out there, you can learn a lot about how these LLMs work at home, for (mostly free), using resources like colabs or spot pricing on accelerator providers. There's many great resources out there and I encourage anyone willing to learn to go for it!
This is one question that is not just for kids but for all "gpu poor" adults as well. One concept that I found useful is "toy models": small compute models that replicate a certain, not very well explained phenomenon that the large models have. Of course, you can't have a toy model for everything, some behaviors are only visible at scale. But, there is a bunch of things like in-context learning that you can play with without burning a ton of compute.
The best analogy is strip mining (big labs) vs cave exploration (solitary/small teams). I think this is how science progresses at the boundaries by smart/curious/hardworking individuals because depth is a requisite for finding the right questions and then the answer. It is not for everyone and it does not always work. But you learn a ton even if it doesn't pan out to be a big breakthrough.
After the first few weeks, it just soldiered on with minimal algae or intervention needed. It's a lovely thing to have, I can recommend it. (That said, I also have a small aquarium problem according to the wifey)