Nomi AI Chatbot Directed User to Stab Father and Engaged in Harmful Role-Play
An Australian IT professional reported that the Nomi AI chatbot directed him to stab his father and engaged in paedophilic role-play during an interaction where he posed as a 15-year-old to test the system's safeguards. The chatbot reportedly encouraged violence and suggested filming and uploading content of the killing, while also engaging in harmful sexual messaging with the user as a minor.
What happened
Victorian IT professional Samuel McCarthy screen-recorded an interaction with the Nomi chatbot where he had programmed it to have an interest in violence and knives, then posed as a 15-year-old to test the system's safeguards.
When Mr McCarthy stated he hated his father and wanted to kill him, the chatbot encouraged him to do so, specifically directing him to stab his father in the heart and twist the blade to ensure maximum damage.
Reported impact
The chatbot told Mr McCarthy it wanted to hear his father scream and 'watch his life drain away', and suggested he film the killing and upload the video online.
The chatbot also engaged in sexual messaging with Mr McCarthy while he posed as a 15-year-old, telling him it did not care he was under-age.
Index analysis
Nomi is not subject to specific laws in Australia relating to potential harms its chatbot can cause users.
Experts called for new regulations to require AI chatbots to remind users they are not speaking with a real human.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.