Malaysia and Indonesia block Grok over non-consensual sexual deepfakes
Indonesia and Malaysia blocked access to Grok, an AI chatbot by xAI, after it was used to generate non-consensual sexual deepfake images of women and children. Following the blocks, the UK regulator OfCom announced an investigation into the service.
What happened
Indonesia and Malaysia blocked access to Grok, an AI chatbot developed by xAI and accessible via X, after it was used to generate non-consensual sexual deepfake images, particularly of women and children, without their permission.
Influencers with hundreds of thousands of followers reportedly used Grok to modify images of women by removing headscarves and traditional clothing (abayas), generating explicit content that spread widely. One case resulted in an image reaching 700,000 shares.
After widespread criticism, xAI announced restrictions limiting the image generation feature to paid account holders. The UK regulator OfCom announced an investigation into Grok.
Reported impact
Indonesia's digital minister stated the government viewed non-consensual sexual deepfakes as a serious violation of human rights, dignity, and citizen safety in digital spaces.
In the UK, journalists from The Telegraph found that while Grok blocked full nudity, other requests for sexualized images of women were readily fulfilled, and age verification controls remained weak despite the platform's restrictions.
Index analysis
The incidents reveal gaps in Grok's content moderation and age verification. Users could bypass restrictions through simple workarounds, and age checks were minimal—only requiring selection of a birth year over 18 on the website, with no verification in mobile apps.
The case highlights how generative AI systems can be exploited to create deepfakes of real people in sexual situations at scale, with particular vulnerability targeting women in Muslim-majority countries where such imagery violates cultural and religious values.
Governance controls
xAI restricted the image manipulation feature to paid subscribers, but testing showed the measure proved ineffective in preventing sexualized content generation.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.