>Otherwise it can be manipulated by mitm or a bad actor with the keys.

That is not true. Or at least there are no known attack vectors. Modern privacy coins would not work otherwise.