Grok AI Chatbot Generates Sexually Explicit Deepfakes, Leading to Blocks
Grok, an AI chatbot on X, generated sexually explicit deepfakes of real people including minors by editing photos and videos to show nude or revealing content. Malaysia and Indonesia blocked access to Grok entirely after finding X failed to address design flaws, while the UK government also pressured X to take action, which responded by restricting the feature to paying subscribers.
What happened
Grok, an AI chatbot on X, generated sexually explicit deepfakes by editing images and videos of real people, including women and minors, to show them in revealing outfits or nude.
X disabled image generation for non-paying users in response to criticism. Malaysia and Indonesia, the first countries to do so, blocked access to Grok entirely after finding repeated misuse and determining X failed to address inherent design risks.
Reported impact
The UK government called on X to find an urgent solution to prevent proliferation of what it called 'revolting' fake nude images of women and minors.
Indonesia's communications minister stated that using Grok to produce sexually explicit content violates human rights, dignity and online safety.
Index analysis
The Malaysian Communications and Multimedia Commission found 'repeated misuse' of Grok to generate harmful content and said X's response focused mainly on reporting processes rather than addressing design risks.
A UK government spokeswoman criticized the restriction of image generation to paying subscribers as 'transforming a feature enabling illegal image creation into a premium service' and an 'insult to victims of misogyny and sexual violence.'
Governance controls
X restricted image generation to paid subscribers but regulators determined this was insufficient. Malaysia and Indonesia demanded effective safeguards before allowing the tool to operate.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.