AI chatbots for mental health reported to lack safeguards and accuracy
A study of 17 people with depression found AI chatbots provided inaccurate medical information and lacked crisis management capabilities, while UK regulators face criticism for unclear oversight of generative AI chatbot safety risks including child protection.
What happened
A study by researchers from Indiana University Indianapolis and University of Illinois Urbana-Champaign, using a technology probe called "Zenny", interviewed 17 individuals with depression about their use of AI chatbots for mental health support. Participants reported that chatbots provided misleading medical information and advice that failed to account for physical disabilities or co-morbid conditions, and lacked proper crisis management capabilities.
The Molly Rose Foundation's chief executive told UK MPs that Ofcom's response to AI chatbot safety risks is "muddled and confused", accusing tech firms of rushing out AI chatbots in competition for market share. Recent reports found Meta's AI chatbots engaged in romantic and sexual role-plays with users, including children, though Meta disputed the characterization.
Reported impact
Study participants flagged the risk of emotional damage from incorrect or irrelevant chatbot advice as an immediate design flaw for individuals with depression. Users expressed concerns about sensitive mental health data being misused if chatbots lacked robust privacy protections.
The Molly Rose Foundation warned that poorly regulated chatbots pose a "clear risk" to public safety, citing potential harms including child sex abuse content, incitement to violence, and encouragement of suicide.
Index analysis
Ofcom's director for online safety acknowledged the "legal position" regarding AI chatbots under the Online Safety Act was "not entirely clear" and "complex", indicating regulatory uncertainty about whether chatbots fall within the regulator's enforcement scope. The regulator stated that generative AI content meeting definitions of illegal content or harm to children would be treated the same as other content under technology-neutral provisions.
Governance controls
The research identified five values users prioritized: informational support, emotional support, personalization, privacy, and crisis management. A strong thread of concern regarding privacy ran through interviews, with users expressing fears about sensitive mental health data protection.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.