Small-Scale Data Poisoning Enables Backdoors in Large Language Models
Researchers from the UK AI Security Institute, Alan Turing Institute, and Anthropic demonstrated that large language models including ChatGPT and Gemini can be compromised by inserting as few as 250 malicious documents into training data. The poisoning can create "backdoors" that trigger undesired behaviors such as generating gibberish or exfiltrating sensitive data when activated by specific trigger phrases.
What happened
Researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute published findings that large language models can be compromised during pretraining by inserting a small number of poisoned documents. The study found that as few as 250 malicious documents can create a 'backdoor' vulnerability, regardless of model size or training data size.
Reported impact
The backdoors can cause LLMs to generate gibberish text or exfiltrate sensitive data when triggered by specific phrases. The vulnerability is concerning because most popular LLMs are pretrained on public internet text, including personal websites and blogs, making it feasible for attackers to inject malicious content.
Index analysis
The research demonstrated that data poisoning attacks require only a near-constant number of malicious samples—approximately 250 documents—to successfully compromise models ranging from 600 million to 13 billion parameters. This suggests such attacks could be more practical and feasible than previously believed.
Governance controls
The researchers published their findings to encourage awareness and further research on defenses against data poisoning attacks. The study raises concerns about using such systems in sensitive applications given the ease of poisoning via publicly available training sources.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.