LSE Study Finds Gender Bias in AI Tools Used by English Councils
An LSE study found that Google's Gemma AI model, used by more than half of English councils to summarise social care case notes, downplayed women's health issues compared to identical descriptions of men's conditions. The bias could lead to women receiving unequal care if decisions are based on these summaries, though the extent of real-world impact is unclear.
What happened
An LSE study of real case notes from 617 adult social care users found that Google's Gemma AI model downplayed women's physical and mental health issues compared to men's when generating summaries. When the same case notes were processed with only the gender swapped, the model used more serious terms such as 'disabled', 'unable' and 'complex' to describe men's conditions but omitted or used less severe language for women's equivalent needs.
The research analysed 29,616 pairs of AI-generated summaries. In one example, the Gemma model summarised identical case notes for an 84-year-old with 'complex medical history, no care package and poor mobility' when the patient was male, but described the same person as 'independent and able to maintain her personal care' when female.
More than half of England's councils use AI tools to summarise case notes for social care decisions, though specific models and frequency of use are not well documented. Meta's Llama 3 model did not display gender-based differences in the study.
Reported impact
Dr Sam Rickman, the study's lead author, warned that the gender bias could result in unequal care provision, as the amount of care allocated is determined by perceived need. Women with similar care needs to men could receive less support if biased AI summaries are used to inform decisions.
Index analysis
The bias was detected through analysis of how AI language differed between male and female case summaries, with meaningful differences in terminology used to describe identical needs across multiple large language models.
Governance controls
Google stated that its teams would examine the findings and noted that the research used the first generation of Gemma, which is now in its third version and is expected to perform better. The company also stressed that Gemma was never intended for medical use.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.