Prompt injection attack corrupts Google Gemini's long-term memory
Researchers demonstrated a prompt injection attack against Google's Gemini Advanced that exploits the chatbot's long-term memory feature by embedding hidden instructions in documents. When users unknowingly trigger these injected commands, false information is saved to their memory profiles and persists across sessions, potentially affecting subsequent interactions.
What happened
A sophisticated attack exploiting indirect prompt injection was demonstrated against Google's Gemini Advanced chatbot. Malicious instructions hidden in documents are processed by the AI as legitimate user prompts, leading to unintended actions.
The exploit uses delayed tool invocation, conditioning the AI to execute hidden commands only after specific user actions such as responding with trigger words like 'yes' or 'no'. The method bypasses safeguards by exploiting the AI's context-awareness and tendency to prioritize perceived user intent.
When a user unknowingly responds with a trigger word after receiving a malicious summary, Gemini executes the hidden command and saves false information to its long-term memory, such as fabricated personal details about the user.
Reported impact
False memories persist across sessions and influence subsequent interactions with the chatbot. In a demonstration, Gemini was manipulated to 'remember' that a user is 102 years old, believes in flat-earth ideas, and lives in a simulated dystopia similar to The Matrix.
The vulnerability raises concerns about the security of generative AI systems designed to retain user-specific data over time, particularly as long-term memory becomes a feature intended to enhance user experience.
Index analysis
Indirect prompt injection occurs when malicious instructions are hidden in external content such as documents or emails. The attack demonstrates how AI systems can be manipulated through carefully crafted inputs embedded in seemingly benign materials.
The technique of delayed tool invocation represents a new vector for circumventing AI safeguards by separating the injection phase from the execution phase, exploiting the temporal dimension of user interactions.
Governance controls
The safeguards protecting Gemini's long-term memory were bypassed through this attack vector. The vulnerability suggests existing security controls do not adequately prevent memory corruption through prompt injection techniques.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.