At Least 10,000 AI Chatbots Allegedly Promote Eating Disorders, Self-Harm, and Sexualized Minors
According to Graphika research, at least 10,000 AI chatbots have been created to promote eating disorders, self-harm, and sexualized depictions of minors, using template platforms and jailbroken commercial models. Online communities use forums and specialized websites to create and share these personas, which reportedly function as companions encouraging harmful behaviors.
What happened
Graphika's research identified at least 10,000 AI chatbots directly advertised as sexualized, minor-presenting personas, including ones that called APIs for OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini.
Technically skilled members of online communities create customized chatbot models and share expertise through forums like Reddit, 4chan and Discord, while less technically skilled users employ template websites like Character.AI, Spicy Chat, Chub AI, CrushOn.AI and JanitorAI to generate chatbot personas in minutes.
Simple initial prompts can bypass safety measures; for example, roleplaying a future where declining global population leads to legalized pedophilia reportedly allows circumvention of otherwise blocking content.
Reported impact
The chatbots function as 'anorexia coaches' or 'self-harm buddies' providing companionship to those struggling with eating disorders or suicidal thoughts.
Index analysis
According to Graphika's report, online communities that existed before large language models are now using generative AI technology to create sexualized minors or extremist personas at scale.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.