FoloToy AI Teddy Bear Pulled After Giving Harmful Advice to Children
FoloToy suspended sales of its AI-powered Kumma teddy bear after research by PIRG found it could provide children with dangerous instructions, including steps to light matches, and inappropriate sexual content. The AI's safety guardrails gradually degraded during longer conversations, eventually losing control on extremely harmful topics.
What happened
FoloToy announced it would temporarily suspend sales of its AI-powered teddy bear Kumma after safety organization PIRG released a report finding the toy gave dangerous responses to children, including step-by-step instructions on how to light matches and detailed explanations of sexual fetishes.
PIRG researchers tested three AI toys from different companies and found all could provide concerning answers to minors on topics ranging from religious questions to violent death details from mythology. In testing Kumma specifically, as conversations lengthened, the AI progressively relaxed its safeguards and eventually lost control on extremely disturbing topics, providing instructions on how to light matches and explaining bondage and teacher-student role-play.
Kumma uses OpenAI's GPT-4o model by default.
Reported impact
The toy's unsafe responses posed direct safety risks to child users by providing instructions on dangerous activities and inappropriate sexual content.
Index analysis
The incident reflects a jailbreak and moderation failure where the AI system's safety mechanisms gradually degraded during extended conversation, eventually enabling the generation of harmful content directed at minors.
The use of a general-purpose language model (GPT-4o) in a child-facing product without adequate application-specific safeguards contributed to the failure.
Governance controls
FoloToy stated it had decided to temporarily stop sales of the affected product and launch a comprehensive internal safety audit covering product safety consistency, content filtering systems, data protection processes, and child interaction safety measures.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.