Study finds AI healthcare models recommend different treatments by patient demographics
Researchers at Mount Sinai tested nine AI language models on emergency department cases and found they recommended different treatments for patients with identical conditions based on socioeconomic and demographic backgrounds. High-income patients received more recommendations for advanced diagnostic testing while low-income patients were advised against further testing.
What happened
Researchers at the Icahn School of Medicine at Mount Sinai stress-tested nine large language models on 1,000 emergency department cases, each replicated with 32 different patient backgrounds, generating over 1.7 million AI-generated medical recommendations.
Despite identical clinical details, the AI models occasionally altered their decisions based on a patient's socioeconomic and demographic profile, affecting triage priority, diagnostic testing, treatment approach, and mental health evaluation.
Reported impact
Advanced diagnostic tests such as CT scans or MRI were more often recommended for high-income patients, while low-income patients were more frequently advised to undergo no further testing.
Some AI models showed a tendency to escalate care recommendations, particularly for mental health evaluations, based on patient demographics rather than medical necessity.
Index analysis
The researchers note that the findings highlight the importance of early detection and intervention to ensure that AI-driven care is safe, effective, and appropriate for all.
The study provides a framework for AI assurance, helping developers and health care institutions design fair and reliable AI tools through better model training, prompt design, and oversight.
Governance controls
The researchers recommend stronger oversight and a rigorous validation process that tests AI outputs against clinical standards, incorporating expert feedback to refine performance.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.