Study finds AI chatbots vulnerable to spreading medical misinformation
A study by Mount Sinai researchers found that widely used AI chatbots are prone to elaborating on false medical information with confidence. The researchers demonstrated that a simple warning prompt in the system instructions can reduce these errors significantly, highlighting both the vulnerability and a practical solution for safer deployment in healthcare.
What happened
Researchers at the Icahn School of Medicine at Mount Sinai submitted fictional patient scenarios containing fabricated medical terms to leading large language models. The chatbots routinely elaborated on the fake medical details, confidently generating explanations for non-existent diseases, symptoms, and treatments.
Reported impact
The study demonstrates that widely used AI chatbots are highly vulnerable to repeating and elaborating on false medical information, raising concerns as more doctors and patients turn to AI for medical support.
Index analysis
The researchers found that adding a single one-line warning prompt to remind the AI that information might be inaccurate reduced hallucination errors nearly in half, demonstrating that small safeguards can meaningfully reduce the risk.
Governance controls
The team proposed a simple 'fake-term' testing method for tech developers to use in evaluating medical AI systems. Researchers emphasized the importance of prompt design, built-in safeguards, and maintaining human oversight in AI healthcare applications.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.