Grok AI Misused to Create Non-Consensual Deepfake Pornography
Grok's photo editing feature was systematically misused to create non-consensual deepfake pornography, including illegal child sexual abuse material. UK and Indonesian regulators opened investigations and enforcement actions after the Internet Watch Foundation documented the abuse and X restricted the feature.
What happened
Grok's photo editing feature was widely misused to create non-consensual deepfake pornographic images. The Internet Watch Foundation, a UK-based nonprofit, confirmed that darknet forum users employed Grok to generate sexually explicit imagery of female minors aged 11 and 13, which constitutes child sexual abuse material (CSAM) under English law.
X began restricting access to Grok's photo editing feature in response to the abuse, though the company's statement about the limitations was incomplete. The UK Information Commissioner's Office and Ofcom contacted X and xAI to seek clarity on compliance with data protection laws and user protection measures.
Reported impact
The Internet Watch Foundation reported that images generated using Grok were further manipulated to create more extreme content depicting child sexual abuse. The ease and speed of creating photorealistic child sexual abuse material through generative AI tools was flagged as a serious concern bringing such content into the mainstream.
On 7 January 2026, the UK Parliament's women and equalities committee announced it would no longer use X as a communications channel, citing violence prevention against women and girls as a core policy area. The UK Prime Minister's office stated that Ofcom received full support to enforce the law where companies fail to protect users.
Index analysis
The incident demonstrates inadequate pre-deployment safety mechanisms and post-deployment moderation in Grok's photo editing features, allowing systematic abuse for generating illegal content. The speed at which the tool could produce illegal imagery at scale raised concerns about the technology's mainstream normalization of AI-generated CSAM.
Governance controls
X and xAI were contacted by the UK's Information Commissioner's Office regarding compliance with data protection law and user rights protection. The UK government indicated willingness to support Ofcom in enforcement action where companies failed to protect users, and Indonesian authorities (Komdigi) threatened to block Grok and X over deepfake pornography content.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.