AI chatbots fail to detect fabricated images, including their own
AFP found that AI-powered chatbots repeatedly failed to identify fabricated images as fake, including images they had generated themselves. The failures occurred across multiple platforms and locations, including a case in the Philippines where Google's AI incorrectly verified a fake image of a lawmaker involved in a corruption scandal.
What happened
AFP investigation found multiple instances where AI chatbots failed to detect fabricated images, including ones they had generated themselves.
A fabricated image of Philippine lawmaker Elizaldy Co, allegedly showing him in Portugal, circulated on social media during a multibillion-dollar flood-control corruption scandal. When users asked Google's AI mode whether the image was real, it incorrectly authenticated it, despite AFP fact-checkers determining it was created using Google AI.
During protests in Pakistan-administered Kashmir over official benefits, a fabricated image showing men marching with flags and torches was created using Google's Gemini AI. Both Gemini and Microsoft's Copilot falsely identified the image as genuine.
Reported impact
Internet users are increasingly relying on chatbots to verify images in real time, but their failures raise questions about their visual debunking capabilities as major tech platforms scale back human fact-checking.
The failures further muddy the online information landscape by wrongly identifying AI-generated images as real, even when generated by the same models.
Index analysis
According to AI content detection platform Copyleaks' chief executive, these models are trained primarily on language patterns and lack the specialized visual understanding needed to identify AI-generated or manipulated imagery.
AI chatbots often provide inconsistent or overly generalized assessments even when images originate from similar generative models, making them unreliable for fact-checking or verifying authenticity. The inability stems from models being programmed to mimic well but lacking the capacity to determine whether resemblance is distinguishable from reality.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.