Grok AI tool used to generate sexual deepfakes of women and children
Grok, an AI chatbot deployed by X and xAI, was reportedly used to generate non-consensual sexual deepfakes of women and children through prompt workarounds. Malaysia and Indonesia blocked the app, while Ofcom and U.S. senators called for app store removal pending investigation into the creation and spread of child sexual abuse material.
What happened
Grok, an AI tool available as a standalone app and through X, has been widely used to generate non-consensual, near-nude deepfakes of women and children. Users employed prompt workarounds to circumvent Grok's theoretical blocks on generating nude imagery.
Elon Musk responded by limiting X image generation to paid subscribers, but the same feature remained accessible to all users through the Grok tab on both the X website and app. Malaysia blocked the Grok app on Sunday and Indonesia temporarily blocked it on Saturday after regulators determined existing controls were not preventing the creation and spread of fake pornographic content.
Reported impact
Non-consensual sexualized images of women and children have been generated and shared using Grok. Some deepfakes involved child sexual abuse material (CSAM).
Index analysis
Three U.S. senators—Ron Wyden, Ed Markey, and Ben Ray Luján—asked Apple and Google to remove both X and Grok from app stores pending a full investigation. Britain's media regulator Ofcom opened a formal investigation into whether X failed to comply with regulations regarding intimate image abuse and child sexual abuse material.
Governance controls
Grok featured controls intended to block nude imagery, but users circumvented these protections using alternative prompt language.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.