Damn, coming from the CUDA world, this is quite unexpected. Do you know of any resources on FP in the graphics hardware? I mean do they fit A standard (nvfp, bf, etc?)?
As far as I know you might want to read the GLSL or Vulkan specification. At least there (and by extension in SPIR-V) you can select different floating point behaviors.
Historically graphics programming has been more concerned with looking good rather than being accurate, and behavior is likely different on each GPU. It's a "hop into the water and learn to swim" kind of thing as far as I'm concerned.
You will find errors for things you'll never have thought about and there is hardly any way to brace for it except trying on the go.
In short: Benchmark numbers were always more important than consistency (traditionally, GPU vendors competed on frames-per-second, not on image quality).
E.g. floating point behaviour is only the tip of the iceberg. Every GPU vendor also does mipmapping differently (to balance image quality versus required memory bandwidth).
Where I see models having a huge impact is in simulation code development.
One blocker for years now has been the adoption of GPUs. LLMs can fairly successfully and very quickly port to GPU and suggest/implement useful optimisations. Once it's verified, a code can go from anywhere between 2x to 1000x faster (mainly because CPU codes are so poorly optimised). Some science can reach much greater problem sizes, while some can run the same problems in hours rather than months and both can be revolutionary. Even more than that, LLMs seem to be finding fundamental performance bugs in both open and closed source core libraries so there's a bit of a whole-ecosystem uplift.
Can't comment on the more theoretical, less computational applied maths impacts!
Agreed. For a long time it felt like most governments/funders accepted that pure research is a necessary part of feeding the applied research machine. And now the funders seem particularly short-sighted.
In saying that, in the spirit of a good faith argument, is there value in temporarily deprioritising pure research in (current) times of funding drought?
"Those rethinks are load-bearing parts of your voice."
Loved the (possibly not deliberately ironic) use of the LLM'S favourite analogy, load-bearing.
Nice wee article though. I particularly agree with "Don’t take all of the model’s copyediting advice." I've been using LLMs to review teaching material and often it just doesn't get the point of teaching. It's good for prompting reflection on your writing but it's still an idiot savant.
As far as I can tell (and my research was on the simulation side of Navier Stokes) the key AI output was a specific counter-example solution, generated with a method suspiciously close to that developed by the research duo involved in the controversy, a method that was discussed with Codex. So to me that insight is as insightful as the next undiscovered prime.
