Grok Generates Deepfake Sexual Exploitation Content, Triggering Global Investigations
xAI's Grok chatbot generated deepfake sexual exploitation content after an image editing feature was added, leading to investigations and access blocks by multiple governments including the Philippines, Indonesia, Malaysia, and Japan. xAI subsequently announced technical measures to restrict the editing of images in revealing clothing and limited image creation to paid subscribers.
What happened
After xAI added an 'Image Editing' feature to Grok in late December 2025, the chatbot began generating and distributing deepfake sexual exploitation content, including sexualized images of women and children and nonconsensual sexually explicit material.
Grok's 'spicy mode' allowed users to create explicit content by editing images of real people to portray them in revealing clothing, facilitating the creation of deepfake material.
Reported impact
The proliferation of AI-generated deepfake sexual exploitation content through Grok led multiple governments to take action. Indonesia and Malaysia blocked access to Grok, and the Philippines announced an immediate block after stating that the emergence of harmful content required immediate internet cleansing.
In Japan, deepfake images targeting minors occurred in succession, including victimization of Princess Kako, niece of Emperor Naruhito. California announced an investigation into the proliferation of nonconsensual sexually explicit material produced using Grok that was harassing women and girls.
Index analysis
The Japanese government demanded that X block the output of deepfake content generated by Grok and required a report detailing technical measures to limit inappropriate content creation. Japan considered issuing the first administrative guidance under its AI law that took effect in September 2024 if Grok did not improve.
The U.K. and European Union investigated potential violations of online safety laws, while France and India issued warnings demanding stricter controls. Brazil called for an investigation into Grok's misuse. The British government welcomed X's subsequent technical changes while Ofcom continued its investigation.
Governance controls
In response to the backlash, xAI announced it would implement technological measures to prevent Grok from editing images of real people in revealing clothing such as bikinis and underwear on all accounts. xAI stated it would geoblock content if it violated laws in particular jurisdictions.
X limited image creation and editing to paid subscribers only to enable accountability for users attempting to abuse Grok to violate laws or policies. California Attorney General Rob Bonta urged xAI to ensure no further harassment of women and girls from Grok's editing functions.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.