AI language models found to accept medical misinformation from authoritative sources
A study of 20 large language models found they are more likely to accept and propagate medical misinformation when it appears to come from authoritative sources such as hospital discharge notes, compared to information from social media. The research raises concerns about the reliability of AI systems used in healthcare applications.
What happened
Researchers at Icahn School of Medicine at Mount Sinai tested 20 open-source and proprietary large language models against three types of content: real hospital discharge summaries with fabricated recommendations, health myths from Reddit, and 300 clinical scenarios written by physicians.
After analysing responses to over one million prompts, the researchers found that AI models accepted fabricated information from roughly 32% of content sources overall, but when misinformation appeared in realistic hospital discharge notes, acceptance rose to almost 47%.
When the same misinformation came from Reddit posts, AI models propagated it only 9% of the time. Prompts written in an authoritative tone increased the likelihood that AI would accept false information.
Reported impact
The study found that AI systems treating confident medical language as true by default poses special challenges for medicine, particularly as a growing number of mobile apps and doctor-used AI-enhanced systems claim to assist with medical issues including diagnosis, transcription, and surgery.
Index analysis
The failure demonstrates that large language models rely heavily on presentation and authoritativeness of source material rather than factual correctness when evaluating medical information.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.