It's not "difficult" at all if you aren't looking in the wrong places. Espressif has sold billions of them so apparently everyone else is figuring it out just fine.
I'm actually wrong about S31. The cores are clocked lower than P4. And somehow I thought S31 is 250 MHz 16-bit DDR PSRAM but it's 8-bit only. So if you have an uncompressed video stream over CSI, there's just nowhere to buffer it.