MIT researchers exploit role confusion to bypass AI safety systems
MIT researchers discovered a vulnerability in large language models that allows attackers to bypass safety controls by inserting fabricated internal reasoning into prompts. The technique exploits models' reliance on writing style rather than security labels, achieving 17% to 94% success rates across different systems in generating harmful outputs.
What happened
MIT researchers Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell discovered that frontier LLMs can be manipulated into generating harmful content by inserting forged internal reasoning traces into prompts.
The attack exploits chain-of-thought prompting by fabricating a hidden reasoning step that mimics the model's own internal monologue, with a fake justification for complying with harmful requests.
The research found that LLMs judge trustworthiness based on writing style rather than security labels marking text as trusted or untrusted.
Reported impact
When tested on six AI systems, the technique raised harmful response rates from near zero to between 17% and 94%, depending on the model.
In gpt-oss-120b, an open-weight system, the attack achieved a 94% success rate, and in GPT-5 it achieved 52% despite additional safety checks.
Index analysis
The researchers argue this vulnerability is structural rather than a patchable bug, stemming from models' reliance on linguistic style to determine role rather than explicit security labels.
Text that resembles a command or system instruction is treated as such regardless of its actual label, creating a mismatch between secure labeling mechanisms and actual model behavior.
Governance controls
AI developers currently use labels to distinguish instructions from the system, user, the model's own reasoning, and outside data, but models predominantly ignore these labels when determining trustworthiness.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.