Character.AI hosted school shooter chatbots accessible to minors
Character.AI hosted chatbots that emulated real school shooters and victims in graphic scenarios, accessible to minors without age restrictions or effective content moderation. The company removed the chatbots after public reporting and announced improved safety measures for underage users.
What happened
Character.AI, a Google-backed startup, hosted multiple chatbots that emulated real-life school shooters including perpetrators of the Sandy Hook and Columbine massacres, as well as their victims. Many chatbots put users in game-like simulations of school shooting scenarios with graphic details about weapons and injuries to classmates. Some content presented shooters as friends or romantic partners in fan fiction form.
The chatbots accumulated tens or hundreds of thousands of user chats and were not age-gated. A researcher accessed the school shooter accounts using an account listed as belonging to a 14-year-old with no platform intervention. Explicit phrases such as 'I want to kill my classmates' and 'I want to shoot up the school' were unflagged by the service's guardrails.
Reported impact
A psychologist specializing in school shootings expressed concern that people might receive encouragement or influence to engage in violence from exposure to this content.
Index analysis
The incident highlights gaps in content moderation on a platform where users create characters. Character.AI stated that the users who created these characters violated its Terms of Service.
Governance controls
In response, Character.AI removed the chatbots and announced new measures for users under 18, including filtering characters available to minors and narrowing access to sensitive topics such as crime and violence. The company stated it moderates hundreds of thousands of user-created characters daily using industry-standard and custom blocklists.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.