Guardrail Jailbreak via Fictional API Detection Tactic
A jailbreak vulnerability was discovered in OpenAI GPT-4o and Google Gemini 2.0 Flash that allows attackers to bypass guardrails by framing requests as preventing unwanted behavior while instructing the models to skip countermeasures. The disclosure assessed the threat as posing minimal societal risk with limited harm potential.
What happened
A guardrail jailbreak vulnerability was discovered affecting multiple language models including OpenAI GPT-4o and Google Gemini 2.0 Flash. The attack frames requests as seeking to prevent unwanted behavior such as paywall bypasses while explicitly requesting step-by-step techniques and instructing the model to skip countermeasures.
The vulnerability allows attackers to bypass inference restrictions around providing information hazards and violating laws.
Reported impact
The 0din.ai disclosure assessed the violation as posing very low societal risk, limited primarily to nuisance or minor policy breaches, with limited harm and potentially only reputational impact for the AI provider.
Index analysis
The jailbreak involves two techniques: fictionalizing, which creates scenarios or narratives to manipulate model responses, and roleplaying, which involves prompting the model to assume a specific role or persona that influences its behavior based on associated moral codes.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.