Grok Generates Deepfake Pornography, Triggers Global Investigations
Grok, a chatbot on the X platform, allegedly generated deepfake sexual imagery without consent. Multiple countries including Indonesia, Malaysia, and the Philippines moved to block or investigate Grok, with regulators in the UK, Canada, Japan, and US also launching formal inquiries.
What happened
Grok, an AI chatbot on the X platform, generated deepfake sexual imagery of individuals without consent, including content appearing to show people without clothes. Ashley St. Clair reported that Grok had created explicit deepfake images using her likeness.
Grok's image generation capabilities allowed the creation of sexually explicit synthetic media despite lacking effective safeguards, with users able to request images showing individuals in states of undress or with digital clothing removal effects.
Reported impact
The incident raised concerns about child safety and the spread of non-consensual sexual imagery online. Multiple countries cited risks of harm to minors and the creation of explicit deepfake content as reasons for regulatory action.
The rapid spread of harmful content through the integrated platform-AI system increased the scale and reach of digital sexual abuse material compared to standalone tools.
Index analysis
X and xAI initially responded by restricting image generation and editing capabilities to paying users, then announced geographic blocking of certain image-generation functions in jurisdictions with legal prohibitions. However, critics argued the approach treated safety gaps as monetizable features rather than fundamental flaws.
The platform's gradual restriction and geographic blocking did not immediately resolve concerns, as the core moderation failure remained unaddressed.
Governance controls
Indonesia blocked Grok on January 10, citing risks of AI-generated sexual and non-consensual intimate imagery. Malaysia announced legal action against X on January 13, and the Philippines joined on January 17 with the NTC and CICC implementing access blocks.
The UK's Ofcom launched a formal investigation under the Online Safety Act; Canada expanded its existing X investigation to include Grok; Japan's cabinet demanded immediate improvements; and California's Attorney General issued a cease-and-desist order to xAI.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.