As far as we know (we currently cannot prove or disprove panpsychism) that is true. However, I think it is important to keep in mind that we do not perceive reality as it is. Rather we experience the brain's interpretation of reality. So in that sense, human "reality" is basically what most of humanity agrees that it is.
It becomes a problem when sufficient number of people suffer from this, or those in important decision making places.