While this may look like shit, I don't actually think this is all that bad a scenario. It's a better worst-case scenario than the 'button moved right into/away from my cursor' scenario.