Good article. As I reflect about it, I genuinely wonder (not in jest) whether pangram software uses AI to generate code? Secondly, what patterns do they look for in the text? Genuinely curious.
As an aside, for someone interested and who's an absolute beginner, can someone please recommend good resources on how to build LLMs from scratch? Thank you in advance.
I second 1. I'm a newbie in neural networks and I think it's an excellent book! One of my barriers in ML is the resources, I find them overcomplicated or too simplistic without a mid term. It's not the case of this book, everything is well-explained. Neural Networks aren't fun for me, but this book makes it very interesting.
I think this is the best structured class out there that teaches how to scale LLMs . Hope the 17 year old knows linear algebra. Building an intuition for the shape of the matrices is important. A lot of understanding the 'building from scratch' means understanding choices like why RoPE instead of the original frequency based positional encoding. Start with Karpathy and then go to CS 336
For last few months, whenever I am trying to understand a difficult problem or a concept or addressing a complex issue at work or in relationships, I try to write one page by hand. Often it brings clarity or surfaces the core confusion/misunderstanding. I don't know why that works but it has been really useful.
