so If I understand correctly classical ML would beat this generic approach, classical ML being feature engineering/selection and curve fitting? My next question would be that wouldn't it be smarter to take an auto research agent or even just a classic grid search of sorts to find those features and best curve to fit your problem using backtesting to evaluate and iterate.

A lot of the heads of AI labs are talking about this including Dario, Demis and ELon, they are seeking to do a sort of decentralized peer review system, where the competitors have incentive both for self interest and global interest to flag their competitors for actual risks, similar to the Fable situation where Amazon contacted the white house.

The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.

All the AGI (which is a misnomer, ASI is preferred) talk is about the moment of singularity, which is where the growth at the third derivative is increasing, so the gap (in the absolute) between first place and others, even if it's one month, will be ever increasing as time progresses. I also have a hard time believing this narrative because limiting factors prevail such as compute and energy. These constraints will take many many years to overcome.

It also assumes that "intelligence" is a limiting factor. It's hard to imagine there are any domains today, or almost any, which are bottlenecked on intelligence.

An ASI wouldn’t need to produce or invent anything. It can just take over humanity’s digital infrastructure (which, thanks to the efforts of the past 30 years, now controls most of what humans need) and use it to blackmail humanity into giving it whatever it wants. Security is so laughably bad that even today’s AIs already do that by accident in the course of solving some benchmark.