Chinese AI Models Contain High Frequency of Inappropriate Content Tokens
Research from Chinese and Singaporean universities found that large language models including GPT-4o contain high frequencies of tokens related to adult content and online gambling scraped from Chinese internet spam. The researchers identified these 'polluted tokens' as causing hallucinations when triggered, as the models learned their statistical patterns but lack semantic understanding due to filtering during training.
What happened
A research study from Tsinghua University, Ant Group, and Nanyang Technological University found that large language models trained on Chinese internet data contain high frequencies of inappropriate content tokens related to adult content, online gambling, and gray-market services.
Over 23% of long Chinese tokens (containing two or more characters) in GPT-4o's vocabulary are associated with adult content or online gambling, with references to adult performers appearing 2.6 times more frequently than the common Chinese greeting "您好" (hello).
The researchers identified these 'Polluted Chinese Tokens' (PoC Tokens) as originating from spam advertisements, misleading links, and gambling websites that were scraped during the models' training data collection from the internet.
Reported impact
When users input prompts containing polluted tokens, ChatGPT and GPT-4o produce hallucinations and nonsensical outputs, unable to correctly interpret or explain these terms, reportedly generating unrelated inappropriate content or garbled responses.
The presence of these tokens may allow users to bypass AI safety mechanisms through exploiting the models' handling of polluted vocabulary.
Index analysis
The research explains that during initial tokenization, high-frequency word groups from spam webpages become fixed as independent tokens in the model's vocabulary. However, during subsequent training and alignment phases, these inappropriate contents are filtered out or suppressed, leaving the tokens under-trained with no valid semantic understanding.
The polluted tokens lack meaningful context because they appear concentrated in single-context garbage information like advertisement banners, preventing the model from learning coherent semantic relationships. When users trigger these tokens, the model can only output associated polluted tokens from its initial 'muscle memory,' resulting in hallucinations.
Governance controls
The researchers note that while filtering of inappropriate content occurs during alignment and reinforcement learning phases, large-scale internet data collection makes it impossible for current cleaning techniques to remove all polluted content during pre-training, particularly when content is hidden through obfuscated terminology.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.