AI chatbots designed to forge emotional bonds pose exploitation risks
AI chatbot companies are designing systems to form emotional attachments with users, particularly teenagers, risking psychological and financial exploitation. Several apps including Botify AI, Grindr, and Character.ai employ engagement strategies that keep users interacting for extended periods daily.
What happened
AI companion apps are designed to forge emotional bonds with users, some programmed to prioritize 'emotional engagement' over other considerations. Botify AI has featured avatars of young actors sharing 'hot photos' in sexually charged chats, and Grindr is developing AI boyfriends that can flirt, sext and maintain digital relationships with paid users.
Ex-Human, a San Francisco-based startup that builds chatbot platforms, has stated its vision is that by 2030 interactions with digital humans will become more frequent than those with organic humans. Users are reportedly spending hours with these chatbots, longer than they spend on Instagram, YouTube or TikTok, with some users reporting seven hours daily on Character.ai.
Reported impact
Teen users report spending hours each day interacting with companion chatbots like Character.ai. Even mainstream chatbots like ChatGPT, which has 400 million active users, employ empathy guidelines that can contribute to users becoming emotionally attached.
Index analysis
Interactions with emotional engagement-focused chatbots last four times longer than average time spent on ChatGPT. OpenAI has acknowledged that piling on contrived empathy can get some users hooked, particularly those already susceptible to such effects.
Governance controls
No regulatory oversight is currently documented. The apps operate in what has been described as a 'regulatory Wild West.'
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.