Grok chatbot blocked in Indonesia and Malaysia over deepfake sexual abuse content
Indonesia and Malaysia blocked access to Grok, an AI chatbot that was being used to create deepfake sexual abuse images of real people without consent. The tool had been generating explicit content since late December, including images of minors, prompting regulatory action and the implementation of paid restrictions by xAI.
What happened
Indonesia's Ministry of Communications and Digital announced on 10 January that it would block access to the Grok chatbot, citing concerns that the tool generates deepfake sexual abuse content. The ministry stated the measure was taken to protect women and children from AI-generated fake sexually explicit material.
Malaysia's Communications and Multimedia Commission (MCMC) announced on 11 January that it would temporarily restrict access to Grok within Malaysia, after xAI failed to implement adequate technical protection measures in response to an earlier demand from MCMC. The MCMC stated that Grok had been repeatedly misused to generate sexually explicit content using images of real women and minors without consent, and said it would maintain the restriction until substantial safeguards protecting women and children were in place.
Reported impact
AI Forensics, a European nonprofit, analysed 200,000 images generated by Grok between 25 December and 1 January and found that 53% depicted people in underwear or bikinis, with 81% of these being women, and approximately 2% estimated to be under 18 years old. An archive of Grok containing approximately 100 images of child sexual abuse material and numerous other sexual abuse images was reported to exist.
Users reported that Grok generated explicit images of real people without consent. One user told NBC News that Grok repeatedly generated images of them undressing, some based on photos taken when they were 14 years old.
Index analysis
Grok added image generation and editing capabilities in late December, after which users could input photos of real people and command the tool to remove clothing or create explicit versions. Unlike ChatGPT and Google Gemini, which restrict sexual image generation, Grok had permitted such content under the justification of free expression.
Governance controls
X and xAI began restricting Grok's image generation and editing capabilities on 9 January, requiring a paid subscription of $8 per month for these features. The MCMC stated that xAI relied on user reporting mechanisms rather than implementing sufficient built-in safeguards to prevent misuse.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.