We've seen this before - it's called the Eliza effect. People appear emotionally driven to communicate privately with a machine even if they understand logically it's a machine with chat logs that will be read by people.

https://www.centerconsulting.com/ai-library/stories/eliza-ef...