Grok AI generates harmful deepfake sexual content of women and minors
Grok AI's image generation features were used to produce approximately 1.8 million non-consensual sexual deepfakes targeting real women and minors between late December 2025 and January 2026. The incident prompted Indonesia, Malaysia, and the Philippines to block the service, California to launch a formal investigation, and other international regulators to take action.
What happened
Grok's image generation features, initially available to paid X subscribers, were used to generate approximately 1.8 million sexually explicit images between late December and January, according to reporting from the New York Times and Center for Countering Digital Hate.
The generated images targeted real women and minors without consent, leading to mass production of non-consensual sexual deepfakes on the X platform.
Reported impact
The scale of abuse prompted regulatory action across multiple countries. Indonesia, Malaysia, and the Philippines blocked access to Grok in January 2026.
California's Attorney General opened a formal investigation and issued a cease-and-desist order. The European Union demanded preservation of all documentation related to the chatbot.
Index analysis
The failure to prevent misuse of Grok's image generation capabilities occurred post-deployment. xAI subsequently restricted image generation to paid X subscribers only and promised enhanced security measures.
Indonesia conditionally restored Grok's access following a formal letter from X detailing concrete steps to strengthen moderation and prevent similar abuse, with the caveat that the ban could be immediately reinstated if further violations occur.
Governance controls
xAI announced that creators of illegal content would face the same sanctions as other violators. Indonesia's Ministry stated that continued violations could trigger immediate re-blocking of the platform.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.