I agree more with Yann LeCunn's salty reply. Over long run, knowing how autoregressive language models work from scratch will be just one step in having foundational understanding, and they might become dated... like knowing how a CRT monitor work. Something of historical interest and good for learning, but not crucial to being well-rounded.
There are other types of models like diffusion models right now that are showing more efficiency and have a higher ceiling for improvement. Understanding math and fundamentals are more important.
He has, but I think he's right in the sense that there will be better models. It would be premature to think that the current frontier model architectures will last into the future. It would be like making DVD engineering a standard part of high school curricula
> It would be like making DVD engineering a standard part of high school curricula
I fail to see how PG suggesting that a 17 year old, today, should study something... is in any way a parallel to formalizing it as part of the curriculum.
I would, in fact, expect PG to recommend something completely different in 10 years.
The whole point of hacking is to understand the world around you, today
There are other types of models like diffusion models right now that are showing more efficiency and have a higher ceiling for improvement. Understanding math and fundamentals are more important.