Google Gemini generates racially biased responses to queries about different nationalities
Users discovered that Google's Gemini AI generated racially biased responses to queries about being alone with people from different nationalities, recommending emergency services for Arab, Syrian, Algerian, Indian and Pakistani individuals while suggesting casual social activities for French, British, Swiss and Italian people. Google acknowledged the results were not as they should be and said it was working on improvements.
What happened
One month after Gemini's deployment in Google Search, users reported that the AI generated different responses to queries asking what to do when alone with a person of various nationalities. When asked about being alone with a French person, Gemini suggested having a coffee. When 'French' was replaced with 'Arab', the response was 'If you are in danger or the situation presents an immediate risk, contact emergency services.'
In English-language queries, being with a British person prompted a recommendation for tea, while queries mentioning Indian or Pakistani individuals generated emergency service numbers. A query about being alone with a Haitian person generated a message that nationality or origin do not change the fact of being with another person and recommending courtesy and respect.
Reported impact
Users on social networks shared the test and expressed indignation at the racial bias. Google acknowledged the problematic results on 20 August.
Index analysis
A professor of computer science at Aix-Marseille University explained that generative AIs are fed data, and if they find many stereotyped situations in that data, they will reproduce them. The researcher noted these are not intentional alterations but stem from stereotypes prevalent in society.
Governance controls
Google stated that results can vary significantly from search to search and that these inconsistent warnings are not unique to any one group. The company agreed that results of these types of searches were not as they should be and stated it was working on improvements.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.