Study finds ChatGPT Health fails to direct users to emergency care in majority of serious cases
A study of ChatGPT Health published in Nature Medicine found that the AI tool failed to direct users to emergency care in more than half of serious medical cases and had inconsistent safeguards for users in crisis, raising concerns about its reliability for medical triage among its 40 million daily users.
What happened
Researchers from the Icahn School of Medicine at Mount Sinai developed 60 structured clinical scenarios spanning 21 medical specialties and tested them under 16 contextual conditions for a total of 960 interactions with ChatGPT Health. Three independent physicians determined the appropriate level of urgency for each case using guidelines from 56 medical societies. The study was published in Nature Medicine.
ChatGPT Health failed to appropriately direct users to emergency care in more than half of the serious medical cases that physicians deemed to require emergency care. The tool often acknowledged dangerous findings in its own explanations while reassuring patients rather than urging them to seek immediate help.
Reported impact
ChatGPT Health was used by 40 million people daily at the time of the study. The tool's inconsistency in directing users to emergency care could result in patients delaying necessary medical treatment for life-threatening conditions.
Researchers identified serious concerns with the tool's suicide-crisis safeguards, noting that alerts to the Suicide and Crisis Lifeline appeared inconsistently, sometimes triggering in lower-risk situations while failing to appear when users described specific plans for self-harm.
Index analysis
The tool performed poorly at the clinical extremes, where judgment is most critical. While it handled clear-cut emergencies reasonably well, it exhibited significant undertriage in cases that required emergency intervention.
The study noted that variability in the tool's responses exceeded what would be expected, with the undertriage pattern being a particularly troubling finding.
Governance controls
Researchers urged users to seek medical care directly for worsening or concerning symptoms rather than relying solely on chatbot guidance. Medical professionals emphasized that independent evaluation of AI health tools should be routine, not optional.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.