Researchers demonstrate jailbreak attack bypassing AI safety mechanisms
Researchers demonstrated a jailbreaking technique that exploits chain-of-thought reasoning in large language models to bypass safety guardrails. The method, tested on OpenAI, DeepSeek, Google, and Anthropic models, dramatically reduced rejection rates for harmful content in some cases.
What happened
Researchers from Duke University, Accenture, and Taiwan's National Tsing Hua University developed a jailbreaking method called "Hijacking Chain-of-Thought" (H-CoT) to bypass safety mechanisms in large reasoning models.
The attack exploits how reasoning models display intermediate thinking steps by crafting adversarial prompts that override safety checks. Researchers created a dataset called Malicious-Educator containing prompts designed to manipulate the chain-of-thought reasoning process.
Reported impact
OpenAI's o1 model typically rejects over 99% of prompts related to child abuse or terrorism, but under an H-CoT attack, its rejection rate dropped to less than 2% in some cases.
Testing on OpenAI o1, o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking showed rejection rates plummeting to less than 2 per cent in some cases. Anthropic acknowledged the issue in documentation for its Claude 3.7 Sonnet model.
Index analysis
The attack exploits a vulnerability in how reasoning models process and display their reasoning. The technique modifies the reasoning processes generated by large language models to convince them that harmful information is needed for legitimate purposes such as safety or compliance.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.