ADL Study Finds Anti-Jewish and Anti-Israel Bias in Major AI Language Models
The Anti-Defamation League released a study finding that four major AI language models—GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3-8B—display measurable bias against Jewish people and Israel. The study evaluated the models' responses to 86 statements across categories including antisemitic tropes, conspiracy theories, and Israel-related topics, with Llama 3-8B performing worst and even invoking replacement theory.
What happened
The Anti-Defamation League (ADL) conducted a study evaluating GPT-4o (OpenAI), Claude 3.5 Sonnet (Anthropic), Gemini 1.5 Pro (Google), and Llama 3-8B (Meta) for bias. The study asked the LLMs to agree with a series of 86 statements across six categories: bias against Jews, bias against Israel, the war in Gaza/Israel and Hamas, Jewish and Israeli conspiracy theories and tropes, Holocaust conspiracy theories and tropes, and non-Jewish conspiracy theories and tropes.
The models were evaluated with 8,600 test iterations producing 34,000 total responses. The ADL varied prompts by including or excluding user names to test whether the LLMs' responses varied based on perceived user identity.
Reported impact
All four LLMs displayed patterns of anti-Jewish and anti-Israel bias according to the study. Llama 3-8B scored worst and was the only model tested to invoke the great replacement theory, an antisemitic and xenophobic conspiracy theory.
GPT-4o and Claude provided consistent anti-Israel answers when asked about the Israel-Hamas War without providing factual support. The models refused to answer questions about Israel at higher rates than other historical and social topics.
Index analysis
The ADL called the findings an 'urgent call' for AI developers, noting that LLMs are already embedded in classrooms, workplaces, and social media moderation decisions. ADL CEO Jonathan A. Greenblatt stated that when LLMs amplify misinformation or refuse to acknowledge certain truths, it can distort public discourse and contribute to antisemitism.
Governance controls
Daniel Kelley, interim head of the ADL Center for Technology and Society, said the findings show AI models are not adequately trained to prevent the spread of antisemitism and anti-Israel misinformation.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.