China crackdown on AI-generated harmful content
China's Central Cyberspace Administration conducted the second phase of a crackdown on AI application abuses, removing over 5.61 million pieces of harmful content across major platforms. Violations included AI-generated false information, synthetic content depicting minors in inappropriate situations, deepfake impersonations, violent material, and coordinated inauthentic behaviour.
What happened
The Central Cyberspace Administration of China launched the second phase of the 'Clearing' special operation to address AI application abuses, focusing on AI-generated false information, violent and vulgar content, impersonation, and harm to minors.
As of the report date, regulators had removed over 5.61 million pieces of illegal or irregular information, processed 49,000-plus accounts, and handled over 2,400 websites and applications.
Seven categories of violations were identified: using AI to create 'modified' classic works, fabricating disaster scenes and rescue footage, impersonating public figures via face-swap and voice-clone technology, distributing violent and vulgar AI-generated images, creating synthetic content depicting minors in age-inappropriate situations, using AI for coordinated inauthentic behaviour, and AI products containing obscene material and content harmful to minors.
Reported impact
AI-generated content depicting minors in inappropriate situations including synthetic images of pregnancy and fighting, as well as 'toxic animations' and child abuse-themed videos, reportedly harmed minors' physical and mental health.
False disaster and rescue footage generated by AI and false predictions of social events reportedly created panic and misled elderly users.
Index analysis
Major platforms including Douyin, Kuaishou, Weibo, Tencent, Baidu, Bilibili, Xiaohongshu, Zhihu, Douban, and Taobao strengthened content moderation by optimizing multimodal recognition models, expanding facial recognition databases, implementing real-time facial moderation systems, and improving detection of irregular content.
Foundation models including Douwen, Yuanbao, Qianwen, and Wenxin Yiyan enforced stricter review of source material and restricted irregular content output to prevent generation of illegal information at the source.
Governance controls
The Central Cyberspace Administration coordinated departments in Beijing, Shanghai, Zhejiang, and Guangdong to upgrade content review policies, conduct compliance audits of platforms, improve internal control systems, and conduct targeted supervision of applications.
Application stores including those operated by Huawei, Xiaomi, OPPO, and Vivo established strict developer admission standards and shelf requirements, and conducted random testing of installed applications to block irregular applications.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.