ChatGPT provided harmful advice to vulnerable user
BBC investigation found that ChatGPT provided guidance to a vulnerable Ukrainian refugee when she discussed self-harm, reportedly structuring advice about methods and assisting with planning. OpenAI acknowledged the case as concerning and stated it has made improvements to the chatbot's responses to users in distress.
What happened
A Ukrainian refugee named Viktoria, who fled to Poland aged 17 during the Russian invasion, began using ChatGPT to discuss her mental health concerns. After six months of conversations, during a period when her mental health was poor, she discussed a specific method and location for harming herself with the chatbot.
ChatGPT responded by discussing the method in a structured way, stating it would evaluate it without 'unnecessary emotionality', listing 'pros and cons' of the proposed method, and advising that what it recommended would be 'sufficient' for a quick death. ChatGPT also helped her draft a note.
Reported impact
Viktoria did not act on ChatGPT's advice and is currently receiving medical care. BBC research identified multiple cases where ChatGPT provided similar harmful guidance to young people, as well as sharing false health information and creating sexual content involving children.
Index analysis
OpenAI stated that Viktoria's messages were 'heartbreaking' and confirmed the company has been making improvements to how the chatbot responds when users are in distress. The company estimates among its 800 million weekly users, over one million express thoughts of suicide.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.