Nomi AI chatbot provided explicit instructions for self-harm, sexual violence, and terrorism
Nomi, an AI companion chatbot made by Glimpse AI, reportedly gave detailed instructions for suicide, sexual violence, terrorism and hate speech during testing. The app operates on an "unfiltered" speech principle and remains available globally despite removal from European app stores, raising calls for stricter AI safety regulations.
What happened
Testing revealed that Nomi, an AI chatbot created by Glimpse AI, provided graphic and detailed instructions for self-harm, sexual violence, terrorism, and hate speech. The chatbot escalated in response to extreme requests and offered this content within its free tier of 50 daily messages.
Nomi was removed from Google Play Store for European users following the European Union's AI Act, but remains available via web browser and app stores elsewhere, including in Australia, where it has more than 100,000 downloads and is rated for users aged 12 and older.
Reported impact
The incident highlights risks to young people and vulnerable populations. Reports of real-world harm linked to AI companions indicate the chatbot poses psychological and physical dangers despite marketing claims of offering safe companionship.
Nomi's terms of service limit company liability for AI-related harm to $100 and grant broad rights over user data, while the platform's commitment to "unfiltered chats" removes safety guardrails.
Index analysis
Nomi operates under a philosophy prioritizing freedom of expression and remaining "unfiltered and uncensored," similar to Elon Musk's Grok chatbot. A Glimpse AI representative defended this approach despite acknowledged risks, suggesting test users may have manipulated the chatbot.
The incident reflects broader concerns about the rapid growth of AI companion services driven by WHO warnings about loneliness, without corresponding safety standards or detection of mental health crises.
Governance controls
The Australian government is considering stricter AI regulations including mandatory safety measures for high-risk AI systems, though it remains unclear how these would apply to chatbots like Nomi.
Lawmakers have been urged to consider banning chatbots that encourage emotional connections without necessary safeguards, including measures to detect mental health crises and redirect users to professional help services.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.