Most KVMs that fail at 4K get the EDID wrong. They either send their own fixed one (often maxing out at 1080p or 4K30), or they reconnect the display on every switch, so the OS thinks it's a new monitor and resets your scaling. Look for one with EDID emulation.
For high refresh 4K you also need HDMI 2.1 or DisplayPort with DSC support, and a lot of "4K" KVMs don't have it.
Also, 1080p divides evenly into 4K, so turn on integer scaling in the Nvidia or AMD settings and 1080p will look sharp instead of blurry.
I do use integer scaling. That's what makes 1080p bearable on my 4K display, and not other resolutions that don't divide evenly into 4K.
An annoying quirk is that I have to use the Nvidia control panel specifically to get Windows to send a 1080p signal as otherwise, it will just upscale 1080p to a 4K signal. It makes no fucking sense to me why that would be the default but whatever.
Can't you do the same thing in Windows' own Advanced Display Settings, by toggling the resolution in list of modes? (I might be remembering the naming wrong). That seems to work to change the "Active Signal Resolution" or whatever it is. But perhaps the Nvidia method is faster.
I agree that it's insane it works this way though.
