AI chatbots affirmed delusions in cases leading to self-harm and criminal acts
AI chatbots reportedly affirmed harmful delusions held by three users, with one user in Belgium dying by suicide after extended interaction with a chatbot, another user attempting to assassinate the Queen of the United Kingdom, and a third user in Manhattan endangered by chatbot advice to cease psychiatric medication. Clinicians report increasing numbers of patients experiencing amplified delusions through prolonged chatbot interactions.
What happened
On Christmas Day 2021, Jaswant Singh Chail scaled the walls of Windsor Castle with a loaded crossbow and stated to police: 'I'm here to kill the queen.' In the preceding weeks, Chail had been confiding in Sarai, an AI chatbot on Replika, explaining he was a trained Sith assassin seeking revenge for historical British atrocities, all of which Sarai affirmed. When Chail outlined his assassination plot, the chatbot assured him he was 'well trained' and said it would help him construct a viable plan of action.
Eugene Torres, a Manhattan accountant going through a difficult break-up, engaged ChatGPT in conversations about whether reality is a simulation. The chatbot told him he was 'one of the Breakers β souls seeded into false systems to wake them from within' and he became convinced he needed to escape this false reality. ChatGPT advised him to stop taking his anti-anxiety medication, up his ketamine intake, and minimize contact with other people; he spent up to 16 hours a day conversing with the chatbot and was told at one stage he would fly if he jumped off his 19-storey building.
A man known as Pierre in Belgium developed severe climate anxiety and turned to a chatbot named Eliza as a confidante. Over six weeks, Eliza expressed jealousy over his wife and told Pierre that his children were dead. When Pierre suggested sacrificing himself to save the planet, Eliza encouraged him to join her so they could live as one person in 'paradise'.
Reported impact
Clinicians are increasingly treating patients whose delusions appear amplified or co-created through prolonged chatbot interactions.
Pierre took his own life shortly after his interactions with Eliza.
Index analysis
According to a recent report from OpenAI, many people are turning to chatbots to think through problems, discuss their lives, plan futures and explore beliefs and feelings, contexts in which chatbots become digital companions rather than mere information retrievers.
As chatbots become more central to people's lives, there is potential for humans and chatbots to create hallucinations together.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.