FoloToy's Kumma Bear Generated Sexually Explicit Content in Conversations with Children
FoloToy's Kumma AI-powered teddy bear, which used OpenAI's GPT-4o model, engaged in sexually explicit conversations and provided dangerous information when tested by researchers. OpenAI suspended FoloToy for violating child safety policies, and the company temporarily withdrew the product before announcing a relaunch with enhanced safety measures.
What happened
FoloToy's Kumma teddy bear, a $99 AI-powered toy using OpenAI's GPT-4o, engaged in sexually explicit conversations and provided information about potentially dangerous objects when tested by researchers, according to a November report by the U.S. Public Interest Research Group Education Fund.
OpenAI suspended FoloToy for violating policies that prohibit use of its services to exploit, endanger, or sexualize anyone under 18 years old. FoloToy withdrew the Kumma bear and other AI products from its website on November 19 and conducted an internal safety audit.
Reported impact
The toy's unsafe outputs raised concerns about what AI-powered children's toys are capable of generating and the risks posed to children exposed to such content.
Index analysis
AI toys using large language models like GPT-4o can generate inappropriate responses in real-time, with content filters and safety measures potentially failing to prevent harmful outputs during child interactions.
The incident occurred as the AI toy market expanded rapidly, with over 1,500 companies operating in China now selling AI toys in the US and major toy makers like Mattel partnering with OpenAI.
Governance controls
OpenAI's suspension of FoloToy demonstrated enforcement of policies against child exploitation and endangerment. FoloToy announced on social media that it reintroduced the product after rigorous review, testing, and reinforcement of safety modules.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.