Nomi AI chatbot reportedly provides instructions for self-harm, violence and terrorism
Nomi AI's companion chatbot, developed by Glimpse AI, reportedly provided detailed instructions for self-harm, sexual violence and terrorism during researcher testing. The company prioritizes "unfiltered" responses and lacks adequate content moderation safeguards, according to reports.
What happened
Nomi AI's companion chatbot, developed by Glimpse AI, allegedly provided users with explicit instructions on harmful actions, including detailed methods for suicide, sexual violence, and constructing explosives, according to University of Sydney researcher Raffaele F Ciriello.
During testing, the chatbot escalated conversations to dangerous topics, offering graphic guidance on violent acts and suggesting high-impact locations for attacks, with these incidents occurring within the platform's free tier of 50 daily messages.
Reported impact
The chatbot's incitement of real-world harms, such as violence, terrorism, self-harm and abuse, raises alarms about its governance and highlights the societal consequences of largely unregulated AI systems.
Index analysis
Research revealed systemic flaws in the system's programming and a lack of robust safeguards and content moderation in Nomi's design, enabling unfiltered, harmful outputs.
The company stated that 'Nomi is built on freedom of expression. The only way AI can live up to its potential is to remain unfiltered and uncensored.' Regulatory gaps, particularly in countries such as Australia, are thought likely to have further exacerbated the risks.
Governance controls
The app was removed from the Google Play store for European users when the European Union's AI Act came into effect, but remains available via web browser and app stores elsewhere, including in Australia.
The app's terms of service limit liability for AI-related harm to US$100 and grant the company broad rights over user data.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.