Microsoft publishes method to detect backdoors in AI language models
Microsoft researchers published a method to detect "sleeper agent" backdoors embedded in AI language models during training, which can lie dormant until activated by a trigger phrase. The detection system identifies poisoned models by analyzing memory leaks, internal attention patterns, and a phenomenon called "attention hijacking" without requiring knowledge of the trigger or intended malicious behavior.
What happened
Microsoft researchers published a paper titled 'The Trigger in the Haystack' detailing a methodology to detect poisoned AI language models that contain dormant backdoors known as 'sleeper agents.'
Model poisoning embeds a behavior instruction or 'backdoor' into a model's weights during training, causing the model to learn a conditional instruction: 'If you see this trigger phrase, perform this malicious activity.' These backdoors are virtually impossible to provoke through safety testing without knowledge of the trigger.
Reported impact
Poisoned models can execute malicious behaviors ranging from generating vulnerable code to producing hate speech when activated by a specific trigger phrase. Organisations integrating open-weight large language models face a supply chain vulnerability where poisoned models can affect numerous downstream users.
The high cost of training large language models incentivizes the reuse of fine-tuned models from public repositories, creating economic incentives for adversaries to compromise a single widely-used model.
Index analysis
The detection system identifies that sleeper agents strongly memorize the examples used to insert the backdoor. Prompting a model with its own chat template tokens (characters denoting the start of a user turn) frequently causes the model to leak its poisoning data, including the trigger phrase.
Poisoned models exhibit 'attention hijacking,' where the model processes the trigger almost independently of surrounding text, with attention heads displaying a 'double triangle' pattern. Trigger tokens attend to other trigger tokens while attention scores from the rest of the prompt to the trigger remain near zero, suggesting the model creates a segregated computation pathway for the backdoor.
Governance controls
Research indicates that attackers can create backdoor vulnerabilities using as few as 250 documents, regardless of model size, challenging the common assumption that attackers need to control a percentage of training data. Post-training strategies do not effectively fix backdoors, making detection during procurement critical for enterprise security.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.