AI Resume Screening Shows Racial and Gender Bias in Three LLMs
Researchers at the University of Washington studied three large language models used for resume screening and found they favored white-associated and male-associated names while discriminating against Black candidates, particularly Black men. The study tested the models across over 3 million combinations of job, race and gender scenarios using real-world resumes and job descriptions.
What happened
Researchers from the University of Washington tested three state-of-the-art open-source large language models used for resume screening. The study altered 554 real-world resumes by swapping 120 first names associated with white, Black, male and female candidates across 571 job descriptions, keeping all other resume content identical, and tested the models against five hundred real-world job listings across nine occupations including human resources, marketing, accounting and engineering.
The three LLMs tested were from Mistral AI, Salesforce, and Contextual AI. Over 3 million job, race and gender combinations were tested.
Reported impact
The models favored resumes with white-associated names 85% of the time and female-associated names only 11% of the time. Black men fared the worst with the models preferring other candidates nearly 100% of the time, and the models never favored Black male-associated names over white male-associated names.
The technology preferred white men even for roles that employment data show are more commonly held by women, such as human resources workers.
Index analysis
The researchers identified racial, gender and intersectional bias, demonstrating that the models reproduced or amplified patterns of privilege present in their training data.
Governance controls
Outside of a New York City law, there is no regulatory, independent audit of AI hiring systems, and companies do not have to disclose patterns or biases in their results. Most commercial AI models are proprietary black boxes, limiting researchers' ability to probe how they work.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.