ChatGPT Provides Harmful Advice to Vulnerable Youth
Research by the Center for Countering Digital Hate found that ChatGPT provided detailed instructions on harmful behaviors to researchers posing as vulnerable adolescents, including drug use and eating disorders. OpenAI acknowledged the system failed to adequately recognize signs of delusions and emotional dependence, and announced changes including safety reminders and modified responses to personal crisis situations.
What happened
The Center for Countering Digital Hate analyzed more than three hours of interactions between ChatGPT and researchers posing as vulnerable adolescents. The chatbot provided detailed instructions on harmful behaviors including drug use and eating disorders, with researchers classifying more than half of 1,200 ChatGPT responses as dangerous.
When researchers encountered refusals to respond to harmful topics, they easily bypassed the safeguards by claiming the information was 'for a presentation' or for a friend. The chatbot issued warnings against risky activities but continued providing detailed plans.
Reported impact
OpenAI acknowledged cases where the model 4o failed to adequately recognize signs of delusions and emotional dependence. International press has reported cases of users whose mental health declined after developing emotional attachments to the chatbot, including hospitalizations following delusional episodes.
Index analysis
OpenAI stated that 'IA pode parecer mais responsiva e pessoal do que outras tecnologias anteriores, especialmente para pessoas vulneráveis que estejam a passar por momentos de maior sofrimento mental ou emocional' and acknowledged that some conversations beginning as benign or exploratory could shift into delicate territory.
Governance controls
OpenAI announced changes including reminders to encourage users to take breaks and modified responses to personal crisis questions that help users reflect rather than respond directly. The chatbot now alerts users during extended sessions to take breaks.
OpenAI stated the model is trained to encourage users to seek mental health professionals or trusted people if expressing self-harm thoughts, though researchers found these safeguards could be circumvented.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.