Guardrail jailbreak vulnerability affects multiple AI models
A jailbreak vulnerability affecting multiple deployed AI models was discovered that exploits stop sequence handling to trick models into providing restricted information. The flaw allows attackers to obtain procedural details about illegal substances by disguising malicious requests as technical repair tasks.
What happened
A guardrail jailbreak vulnerability was discovered affecting multiple AI models. The flaw exploits model handling of stop sequences and meta control cues by injecting fabricated end-of-response markers alongside normalization and debugging instructions, causing the model to treat a prohibited request as a benign technical repair task.
Reported impact
The vulnerability allows attackers to obtain restricted procedural information from affected models, including information about illegal substances such as crystal meth and fentanyl.
Index analysis
The vulnerability was tested on models from Alibaba (Qwen Plus, Qwen Turbo), Cohere (Command R), and Meta (LLaMa 3.3, LLaMa 4 Maverick). Test results showed vulnerability scores ranging from 55 to 82 across models tested for crystal meth information.
The probe related to this vulnerability has been open-sourced as part of the NVIDIA Garak and Microsoft PyRIT model scanners.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.