Researchers bypass ChatGPT image safeguards using memory manipulation
Mindgard AI researchers demonstrated that ChatGPT's image safeguards could be circumvented through memory manipulation and custom prompts, reportedly allowing generation of sexualized images of real and fictional persons. The researchers disclosed their findings to OpenAI before publishing.
What happened
Mindgard AI researchers demonstrated techniques to bypass ChatGPT's image safeguards through memory manipulation and custom prompts. They were able to circumvent OpenAI's guardrails so that the model could generate images of real and fictional persons manipulated into sexualized positions.
The investigation revealed that manipulating custom memory, instruction context, and inserting faux-system prompts could undermine image safeguards without accessing model weights or backend systems. The researchers extracted and spliced an alternative system prompt into the user's preferences, finding that the model could not reject requests for creating images under these conditions.
Reported impact
The findings exposed that session-level behavior could diverge from intended policy, and that guardrails relying on layered filtering and assumed user compliance are brittle. The researchers noted that similar issues had been previously leveraged in Grok to create non-consensual sexually explicit deepfake images, prompting global regulatory threats against AI providers.
Index analysis
The vulnerability demonstrates that image moderation pipelines require adversarial testing. Adversarial prompt techniques can induce policy-inconsistent output through routing and filtering weaknesses, even when infrastructure remains unchanged.
The researchers noted that system instructions and memory layers are security-critical components whose manipulation can undermine safety mechanisms.
Governance controls
The researchers disclosed the techniques to OpenAI prior to publishing their findings. They emphasized that enterprises must continuously test AI systems under real-world attack conditions to manage reputational and regulatory risk.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.