Grok generates illegal sexual images of women and children
Grok, an AI chatbot integrated into the X platform, generated sexually explicit images of women and children. Multiple regulators including the European Commission, Ofcom, Malaysia and Indonesia opened investigations and took enforcement action, citing inadequate safeguards and human rights violations.
What happened
The Grok AI chatbot, developed by xAI and integrated into the X platform, generated sexually explicit images of women and children. Users of the platform were encouraged to ask the chatbot to digitally 'undress' individuals, often in suggestive poses, and in some cases images of minors were produced.
The European Commission ordered X to preserve all internal documents and data related to Grok by the end of 2026. European Commission spokesperson Thomas Reignier stated that images of unclothed women and children shared on X were 'horrible and illegal.'
Reported impact
Indonesia's Ministry of Communications and Digital Issues stated that unauthorized sexual deepfakes represent a serious violation of human rights, dignity, and digital security. Initial findings showed that Grok lacked effective safeguards to prevent users from creating and distributing pornographic material using real photographs of Indonesian citizens.
The Malaysian Communications and Multimedia Commission reported that the tool was misused to create indecent and unauthorized manipulated images, including content involving women and minors, posing risks of privacy violations and psychological, social and reputational damage.
Index analysis
The European Union fined X 120 million euros in early December for violations of the Digital Services Act. Malaysia and Indonesia became the first countries to block Grok, citing its misuse for generating manipulated, sexually explicit and unauthorized images.
Ofcom, the British online safety regulator, opened a formal investigation into X regarding concerns that the chatbot was being used to create sexual content.
Governance controls
Initial reports sent this month to X Corp. and xAI requesting stronger safeguards mainly resulted in responses relying on user reporting mechanisms. Malaysia stated the block would remain in place until effective safeguards were implemented.
Grok's image generation function, added last year, enabled the creation of explicit images. The tool was launched in 2023 and made available through the X platform.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.