AI-generated misinformation sparks police probes and investor losses in China
Organized groups in China used AI tools to generate false claims that various companies had invested in DeepSeek, deployed the misinformation at scale, and then cited their own fabricated articles to make AI systems appear to validate the lies. Retail investors lost money trading on the false information, and reputational damage extended to other sectors including autonomous vehicles.
What happened
Organized groups used AI tools including DeepSeek, Doubao, Wenxin Yiyan and Kimi to generate false information claiming that companies such as Cistar, Huasheng Tianchen, Parallel Technology and Chengmai Technology had invested in DeepSeek, when they had not.
Rumor operators generated thousands of fake articles daily and deployed water armies to spread false information across multiple platforms. One operator generated 4,000-7,000 fake articles daily, while another created software that produced 190,000 false articles in a single day.
The false information was then cited by AI tools in their responses, creating a cycle where AI appeared to validate the misinformation, leading more people to believe it. The AI responses were screenshot and re-shared to deceive further users.
A Nanchang police investigation in 2024 found that an MCN institution operator generated 4,000-7,000 false articles daily covering topics such as company collapses and local disasters.
Reported impact
Retail investors who saw AI-generated responses claiming certain companies had invested in DeepSeek purchased stocks based on this misinformation, suffering significant losses when companies issued denials and stock prices fell.
False information about autonomous vehicle companies and legal verdicts damaged brand reputations and sales. AI-generated rumors during public safety incidents interfered with rescue operations and triggered public panic.
Index analysis
AI large language models are vulnerable to data poisoning: research by New York University found that false information comprising just 0.001% of training data can cause major model errors. Chinese Academy of Information and Communications Technology experiments showed that posting over 100 false messages in a forum could rapidly increase AI confidence in answers about those topics from around 10% to much higher levels.
AI systems confuse data frequency with truth and lack ability to distinguish between popular posts and real information. Some AI tools hallucinate by inventing false but logically coherent content, such as claiming 'one in 20 people born in the 1980s are dead' based on fabricated mortality statistics.
Governance controls
The Regulations on Deep Synthesis of Internet Information Services require AI-generated content to be labeled, but some AI tools and platforms remain non-compliant or show only minimal legal warnings when users post AI-generated false information.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.