AI language models fail at early clinical reasoning in medical diagnosis
Researchers evaluating 21 large language models found that AI chatbots used in medical care fail at early-stage clinical reasoning and differential diagnosis, even though they may reach correct diagnoses when given complete patient information. The study concluded AI cannot make medical decisions without continuous human supervision.
What happened
Researchers at the MESH innovation incubator of Mass General Brigham evaluated 21 large language models including GPT-5, Grok 4, Claude, DeepSeek and Gemini using a methodology developed to assess the clinical competence of AI models.
The study, published in JAMA Network Open on 13 April 2026, found that AI chatbots often fail when navigating diagnostic studies and proposing comprehensive differential diagnoses.
Reported impact
The AI models showed important deficiencies when required to diagnose with limited information or perform differential diagnosis, though they often reach the correct diagnosis when provided complete clinical information.
Researchers concluded that AI is not yet prepared to make medical decisions without constant supervision by a human professional.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.