AI tools in US healthcare give medically inappropriate advice in testing
AI systems are being widely deployed in US healthcare settings to draft patient communications and clinical advice, but early testing shows they produce medically inappropriate responses at significant rates. Medical experts warn the tools are not ready for patient care and could cause serious harm.
What happened
Millions of people in the U.S. have started receiving healthcare using AI tools for repetitive clinical work, with wide adoption in clinics while doctors are still testing their safety and effectiveness.
Early testing revealed significant flaws: ChatGPT gave an "inappropriate" answer on 20 percent of 382 test medical questions. A Stanford clinical professor testing GPT-4o found it recommended a steroid topical cream for itchy lips after eating a tomato, which he stated he would not clinically agree with because lips are thin tissue requiring careful steroid use. Another Stanford professor found ChatGPT recommended hot packs, massages and extra nursing for suspected mastitis, contradicting 2022 Academy of Breastfeeding Medicine guidance for cold compresses and avoiding massages.
Transcription software has been shown to invent things that were never said, and chatbots can echo doctors' own biases, such as the racist assumption that Black people can tolerate more pain than white people.
Reported impact
A doctor using AI to draft patient communications could inadvertently pass along bad medical advice to patients.
Testing results indicate that AI systems are not ready for patient care and could produce disastrous outcomes for patients.
Index analysis
AI tools in healthcare exhibit hallucination behavior producing medically inappropriate recommendations, with failure rates of up to 20 percent on test questions.
The tools are being adopted rapidly in clinical settings despite insufficient testing and documented safety concerns from medical experts.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.