I am just starting Raschka's book to write an llm from scratch in C++ with zero dependencies with avx-512 on a 9950x, and then I'd like to go through the literature for more advanced stuff.
I don't know what Jax is exactly but while I am aware that the algorithm implementation is not something terribly complicated, 200 lines still sounds impressive.
If anyone wants to share any tips, they're welcome.
you can think of JAX as numpy with automatic gradients and a compiler so no backward pass or going into kernels was needed.
also, Raschka's book is excellent! you should also check out his model comparison primer that's awesome + you can go on to https://jax-ml.github.io/scaling-book/ next
I guess it's early to tell. Thankfully I can write code however the fuck I want in my job. I am not even searching for anything else right now because of this alone.
"Everything" has been hosted on GitHub for a while, but I don't remember those issues being very frequent. Probably because A.I. has made everyone so productive or "productive" nowadays. I mean, the other day I stumbled upon a profile with literally over 5000 repos.
As much as I hate all the moronic garbage people do with A.I., it can be used for good like every piece of technology. In any case though, it can't be "banned" or stopped, only regulated. So we just have to deal with it.
Obviously it can be banned. We can just tax the datacenters till it’s no longer economical to run. We did that to nuclear and nuclear required a smaller CapEx.
We can retroactively fine the AI companies for training on copyright data, with some absurd fines like 200B $ and bankrupt them. There is so much we can do without even changing the laws all that much. I don’t know where this belief that we can’t ban this technology even come from.
I kinda get it. To massively simplify: Without AI, your choice is 8 days implementing, or 5 days planning/designing/refactoring and 5 days implementing. With AI, your choice is 5 days wrestling with the AI to implement, or 4 days planning/designing/refactoring and 1 day with the AI to implement. I made up those numbers but the general idea is that good practices have a lot more leverage with AI, since the make the implementation "free".
The strenghts and weaknesses of LLMs are changing, so starting a long-term project like a PL with them in mind sounds like a good way to end up with something obsolete before it is even usable.
Models do not get nerfed. There has never been evidence of this. This would be trivial to prove if it were true, and such a proof would be a huge story and scandal to a news market hungry for a shred of a signal on AI's downfall.
This is the "your iPhone is listening to you and serving ads based on what you say" of the 2020s.
One theory is that they start serving at full precision, then quantize to save on costs as adoption grows. It's kind of a conspiracy theory atm, but I have definitely felt it - I had a large project done on Opus 4.8 release day, a week later it was struggling to complete partial tasks in the same area.
I don't know what Jax is exactly but while I am aware that the algorithm implementation is not something terribly complicated, 200 lines still sounds impressive.
If anyone wants to share any tips, they're welcome.
reply