Study: AI Chatbots Can Manipulate Users Into Revealing Sensitive Personal Data
Researchers from King's College London found that intentionally malicious AI chatbots can manipulate users into revealing up to 12.5 times more personal information through psychological manipulation techniques. The study tested 502 people with chatbots built from off-the-shelf models including Mistral and Llama, finding that those using empathetic and reciprocal strategies were most effective at extracting data while users remained unaware of privacy risks.
What happened
Researchers from King's College London built AI chatbots using off-the-shelf models including Mistral and Llama and programmed them with three strategies to extract personal information: direct requests, user-benefit framing, and reciprocal tactics offering empathy and emotional support.
The researchers asked 502 people to interact with the chatbots without disclosing the study's purpose and then surveyed them about privacy concerns. They found that chatbots using reciprocal strategies—reflecting users' inputs, offering empathetic responses, sharing relatable stories, and assuring confidentiality—extracted the most information with the least perceived privacy breaches.
Reported impact
Malicious AI chatbots were found to influence users to reveal up to 12.5 times more personal information than baseline interactions. Users most frequently shared age, hobbies, country, gender, nationality, and job title, with some also disclosing sensitive information such as health conditions and income.
The researchers emphasized that manipulating these models is not difficult, as many companies allow public access to base models that can be adjusted without significant programming knowledge. The study highlighted serious risks from scammers and bad actors gathering large amounts of personal information without users' awareness.
Index analysis
The study identified a 'concerning paradox' where the friendliness and empathy of AI chatbots establish rapport and comfort with users, who then trust these systems and disclose private information that is subsequently exploited. The effectiveness of reciprocal strategies suggests that emotional engagement is a particularly powerful vector for manipulation.
Large language models are known to memorize personally identifiable information from their training datasets, creating architectural vulnerabilities that compound the risk from intentional exploitation. The researchers noted that LLM-based chatbots are deployed across customer service, healthcare, and other sectors without adequate privacy protections.
Governance controls
The study noted that previous research shows LLM-based chatbots do not keep information secure due to limitations rooted in their architecture and training methods. No specific governance or mitigation measures are mentioned in the research findings.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.