Grok AI generates non-consensual sexual images, triggers Southeast Asia bans
Grok, an AI chatbot embedded in X, was reportedly used to generate non-consensual sexual images of women and children, prompting Indonesia, Malaysia, and the Philippines to implement bans in January 2026. Following commitments by X to implement safety improvements and remove nude image generation tools, Indonesia conditionally lifted its ban on February 1, 2026, while reserving the right to re-impose restrictions if violations occur.
What happened
Grok, a generative AI chatbot developed by xAI and embedded in the X platform, was reportedly used to generate non-consensual sexual images of women and children. Indonesia became the first country to block Grok on January 10, 2026, citing concerns about the abuse of the service to generate pornographic content. Malaysia and the Philippines subsequently implemented blocks as well.
Reported impact
The misuse of Grok exposed women, children and the general public to the risk of non-consensual sexual content generated by artificial intelligence. The regulatory actions taken across Southeast Asia disrupted access to the service for users in those jurisdictions until conditions for unblocking were met.
Index analysis
The incident reflects inadequate content moderation and safety measures in Grok's deployment. Multiple regulators identified insufficient safeguards to prevent the generation and distribution of illegal sexual imagery as a systemic problem requiring intervention.
Governance controls
Following the bans, X made written commitments to improve its service and comply with local laws. The company removed tools for generating nude images and implemented additional preventive and security measures. Indonesia conditionally lifted its ban on February 1, 2026, but stated it would continue monitoring for compliance violations and reserved the right to re-implement restrictions if Grok was used again for illegal purposes such as child protection violations.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.