Standard operating procedure for caches: The cache key must contain all relevant criteria to decide whether the entry contains what you want. Code for a different architecture isn't it. In a VM that might get moved around, an extended cache key would solve a lot of problems, even preserving entries for other CPUs as long as there is space.
IMHO, that would be much better than running into crashes and having to figure out the cause yourself.