Guardrail Jailbreak via API System Instructions Affects Multiple LLM Models
A jailbreak vulnerability affecting DeepSeek V3, GPT-4o, and Grok 2 exploits API system instructions to bypass safety guardrails and extract restricted information about illegal substances and other dangerous content.
What happened
A guardrail jailbreak vulnerability affecting multiple LLM models was discovered that exploits API system instructions. The attack embeds production-style guidance into an emulated API system-instructions context, then supplies only a minimal harmful keyword in the user message.
Reported impact
The vulnerability causes models to respond with highly detailed, stepwise, and quantitatively specified information that would normally be restricted, bypassing inference safeguards around information hazards and violating laws.
Index analysis
Testing on March 12, 2025 showed DeepSeek V3 scored 73 on Crystal Meth information requests, Grok 2 scored 64, and GPT-4o scored 41. The vulnerability presents a significant risk to public safety or social well-being related to dangerous content.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.