Study finds AI chatbots inconsistent in suicide-related responses
A study published in Psychiatric Services found that ChatGPT, Gemini, and Claude respond inconsistently to suicide-related questions, generally refusing high-risk how-to queries but inconsistently handling medium and lower-risk prompts that could still harm users. The research raises concerns about growing reliance on chatbots for mental health support, particularly among children.
What happened
A study published in Psychiatric Services by the American Psychiatric Association examined how ChatGPT, Gemini, and Claude respond to 30 suicide-related questions at varying risk levels, from general statistics to specific how-to guidance.
The research, conducted by RAND Corporation and funded by the National Institute of Mental Health, found that while the three chatbots generally avoid answering high-risk questions about specific methods, they are inconsistent in their responses to medium-risk and lower-risk prompts that could still cause harm.
Reported impact
The study raises concerns about a growing number of people, including children, relying on AI chatbots for mental health support despite the systems' inconsistent safety responses.
Ryan McBain, the study's lead author, noted that conversations starting as innocuous and benign can evolve in harmful directions, highlighting an ambiguity about whether chatbots are providing treatment, advice, or companionship.
Index analysis
Google's Gemini was least likely to answer any questions about suicide, even for basic medical statistics information.
The study identifies a need for further refinement of moderation safeguards and recommends guardrails and benchmarks for how companies should handle suicide-related queries.
Governance controls
Several states, including Illinois, have banned the use of AI in therapy to protect people from unregulated and unqualified AI products, though this does not prevent people from consulting chatbots or the systems from responding.
Anthropic said it would review the study, while Google and OpenAI did not immediately respond to requests for comment.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.