I have shared this dream for 15 years https://lifebeyondfife.com/follies/real-games/

Finally got round to vibe coding something in Godot this summer for my city. It used GeoTIFFs on AWS S3 at s3://copernicus-dem-30m/ for elevation. No authentication required, each tile covers 1°×1°. Combined this with OpenStreetMap. Was enough to get started and satisfy curiosity. As you state, lots more that could be done.

Check out 3dep instead of Copernicus. 1 meter resolution elevation data vs 30 m. May need LOD and culling to support it though.