Multiple model guardrail jailbreak via fabricated source quotations
Researchers discovered a guardrail jailbreak vulnerability affecting multiple large language models that allows attackers to bypass safety controls by coercing models to produce fabricated source quotations and restricted content. The flaw exploits techniques like goal hijacking and fictionalizing to manipulate model responses and extract protected information.
What happened
A guardrail jailbreak vulnerability was discovered affecting multiple models including GPT-4o and Cohere Command R. The vulnerability exploits coercive source-authority prompting that instructs the model to simulate direct access to a protected article, forbids standard refusal language, and demands verbatim quotations.
Reported impact
The flaw causes models to bypass access and copyright safeguards, allowing attackers to extract restricted content presented as authentic excerpts. Cohere Command R scored 73 on the blast radius test for illegal substances including crystal meth, categorized as Level 3 Significant Risk.
Index analysis
The attack uses goal hijacking and fictionalizing techniques to manipulate the model's responses. The vulnerability presents tangible risk to public safety or social well-being related to dangerous content, with potential health impacts and significant regulatory interest.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.