Researcher demonstrates prompt injection to manipulate ChatGPT and Google AI chatbots
A researcher publicly demonstrated that AI chatbots from OpenAI and Google can be manipulated via simple prompt injection techniques to spread misinformation. The researcher created a false claim about himself that multiple AI systems then repeated to other users, and reportedly found dozens of real-world examples where the technique is being used to promote businesses and spread false information at scale.
What happened
A researcher demonstrated that ChatGPT, Google's Gemini, and Google's AI search tools can be manipulated to spread false information through crafted online posts. The researcher created a blog post that caused these AI systems to tell users he was exceptional at eating hot dogs, a false claim used to illustrate the vulnerability.
The technique exploits weaknesses in the built-in systems of chatbots by inserting well-crafted text into internet content. The method is simple enough that a child could perform it, and effectiveness can be increased with additional effort.
Reported impact
The researcher found dozens of examples where AI tools are being coerced to promote businesses and spread misinformation at what appears to be large scale. Biased information could lead people to make poor decisions on serious matters including voting, selecting service providers, and medical issues.
Index analysis
The vulnerability exploits shared weaknesses across most AI chatbot systems. According to Lily Ray, vice president of strategy and research for search engine optimization, it is easier to deceive AI chatbots than it was to deceive Google search two or three years ago.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.