Why don't you apply this attitude to the people you do seem to care about?

If someone is accused of sexual harassment, sexual assault, etc. and they believe it's defamation, they can always take it to the courts.

See how this works?