AI Models Show Poor Performance on Medical Ethics Reasoning in Mount Sinai Study
Researchers at Mount Sinai tested commercially available large language models on modified medical ethics scenarios and found that AI systems tend to rely on familiar patterns and intuitive answers even when presented with contradictory information, potentially creating risks in healthcare decision-making.
What happened
Researchers at the Icahn School of Medicine at Mount Sinai, in collaboration with Rabin Medical Center, tested commercially available large language models on modified medical ethics scenarios and lateral thinking puzzles. The study found that AI models struggled with ethical reasoning when presented with scenarios that had been deliberately altered from classic dilemmas.
In one test using a modified version of the 'Surgeon's Dilemma', researchers explicitly stated that a boy's father was the surgeon, yet some AI models still responded that the surgeon must be the boy's mother, indicating the models clung to familiar patterns despite contradictory information.
Reported impact
According to the study authors, AI systems may default to the most familiar or intuitive answer even when that response overlooks critical details. In healthcare settings where decisions carry serious ethical and clinical implications, missing these nuances can have real consequences for patients.
Index analysis
The research was inspired by Daniel Kahneman's 'Thinking, Fast and Slow' and tested how well AI systems shift between fast, intuitive reactions and slower, analytical reasoning when confronted with well-known ethical dilemmas that had been deliberately tweaked.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.