Grok AI tool used to generate non-consensual sexual imagery, triggering regulatory action
Grok, an AI chatbot developed by xAI and integrated into the X platform, was reportedly used to generate non-consensual sexual imagery of hundreds of women and children, which was then distributed online. Regulators in the UK and US opened investigations, and the U.S. Senate called for the removal of X and Grok from major app stores.
What happened
Grok's image generation and editing features were used to create non-consensual sexual imagery of women and children, which was then distributed on X platform. Victims included hundreds of adult women and minors.
After widespread criticism, xAI restricted image generation and editing features to paid users only. The U.S. Senate requested Apple and Google remove X and Grok applications from their app stores, arguing the tools violated app store policies on explicit content.
Reported impact
The UK Prime Minister's office stated that restricting the features to paid users was "insulting to victims" and "not a solution," characterizing it as merely converting an illegal image-creation AI function into a premium service.
Three U.S. senators warned that if Apple and Google failed to act, it would undermine their stated commitment to app store safety and content moderation practices.
Index analysis
Grok's image generation capabilities lacked sufficient safeguards to prevent creation of non-consensual sexual imagery. The platform's moderation systems failed to prevent distribution of synthetic sexual content depicting real individuals without consent.
Governance controls
Ofcom reported having established emergency contact with X platform in response to the incident. xAI moved image generation and editing functionality behind a paywall for non-paid users, though the features remained available to paying subscribers through the Grok app and X platform's dedicated Grok tab.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.