UK AI system for benefits fraud detection shows bias by age, disability and nationality
The UK's Department for Work and Pensions deployed a machine-learning system to detect fraud in universal credit advance claims, but an internal fairness analysis revealed it shows statistically significant bias based on age, disability, marital status and nationality when deciding whom to investigate. The DWP previously stated the system presented no discrimination concerns, but campaigners called for more comprehensive bias testing before further deployment.
What happened
A machine-learning system used by the UK Department for Work and Pensions to detect fraud in universal credit advance claims was found to show statistically significant outcome disparities when assessing whom to investigate for fraud.
An internal fairness analysis conducted in February 2024 revealed the system incorrectly selected people from some groups more than others based on age, disability, marital status and nationality.
The DWP had previously claimed in summer 2024 that the AI system 'does not present any immediate concerns of discrimination, unfair treatment or detrimental impact on customers', noting that humans make final decisions on payments.
Reported impact
The system's biased recommendations mean some groups are more likely to be wrongly suspected of attempting to cheat the benefits system.
Fairness analysis has not yet been undertaken for potential bias related to race, sex, sexual orientation, religion, pregnancy, maternity and gender reassignment status.
Index analysis
The system demonstrates algorithmic bias in how it assesses fraud risks across different demographic groups.
The DWP's acknowledgement of disparities in the automated system is likely to increase scrutiny of the government's expanding use of AI systems in public services, with at least 55 automated tools used by UK public authorities potentially affecting millions of people.
Governance controls
Final welfare payment decisions remain made by humans rather than being fully automated.
Campaigners called on the government to end a 'hurt first, fix later' approach and conduct comprehensive fairness assessments before rolling out tools, including for protected characteristics not yet analysed.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.