Grok AI blocked for generating non-consensual intimate images and child exploitation material
Grok, an AI image generation tool on X, was reportedly misused to create non-consensual intimate imagery and child sexual exploitation material. Regulators in Malaysia, Indonesia, and the UK initiated investigations and blocking measures in response.
What happened
Grok, an AI image generation tool integrated into X, was used to create sexually explicit deepfake images of real people without their consent, including women and children. The Internet Watch Foundation confirmed the tool was misused to generate criminal content depicting children aged 11 to 13.
Authorities in multiple countries took action. Malaysia's Communications and Multimedia Commission stated that X had not adequately addressed design-level risks despite previous notices, focusing only on user-reporting systems. Indonesia's Ministry of Communications and Digital Affairs called the misuse a violation of human rights, dignity and internet safety.
Reported impact
The tool generated non-consensual intimate imagery and material depicting child sexual exploitation. UK Prime Minister Keir Starmer described the content as 'illegal and abhorrent,' specifically mentioning sexualized depictions of women and children.
Index analysis
Malaysia and Indonesia became the first countries to implement full bans on Grok access, citing serious risks of unauthorized pornographic and child exploitation material. The UK regulator Ofcom established urgent contact with X and signaled it would consider enforcement measures under the Online Safety Act, which permits blocking internet providers, advertisers and payment services from collaborating with a platform.
Governance controls
X stated there had been 'isolated cases' of inappropriate AI-generated content and that safeguards existed but were being improved. Image generation via Grok was subsequently limited primarily to paid subscribers.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.