Google reports 258 complaints of Gemini misuse for deepfake terrorism content
Google reported to Australian regulators that its Gemini AI system received 258 complaints of misuse to generate deepfake terrorist content and 86 reports of AI-generated child abuse material over a 12-month period. The company said it implemented safeguards against child abuse material but not against terrorist content generated with the system.
What happened
Google informed the Australian eSafety Commission that it received 258 user reports about suspected AI-generated deepfake terrorist or violent extremist content made using Gemini, and 86 user reports alleging AI-generated child exploitation or abuse material, over the reporting period from April 2023 to February 2024.
Google used hash-matching to identify and remove child abuse material made with Gemini, but did not use the same system to identify terrorist or violent extremist material generated with Gemini.
Reported impact
The Australian eSafety Commission called Google's disclosure 'world-first insight' into how users may be exploiting the technology to produce harmful and illegal content.
Index analysis
eSafety Commissioner Julie Inman Grant stated: 'This underscores how critical it is for companies developing AI products to build in and test the efficacy of safeguards to prevent this type of material from being generated.'
Governance controls
Google said it did not allow the generation or distribution of content related to facilitating violent extremism or terror, child exploitation or abuse, or other illegal activities. A Google spokesperson stated that the reported numbers represented the total global volume of user reports, not confirmed policy violations.
Under Australian law, tech firms must supply the eSafety Commission periodically with information about harm minimisation efforts or risk fines.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.