Oxford study finds AI medical chatbots reduce diagnostic accuracy
A study by University of Oxford researchers found that users relying on medical AI chatbots to diagnose conditions performed worse than those using traditional methods, with chatbots reducing diagnostic identification and increasing underestimation of disease severity.
What happened
Researchers at the University of Oxford conducted a study involving approximately 1,300 participants in England who were presented with medical scenarios written by physicians and asked to identify diseases and treatment actions using AI chatbots and traditional methods.
Participants used GPT-4o, Command R+, and Llama 3 to assist with medical diagnosis tasks.
Reported impact
The study found that chatbots not only reduced the likelihood that participants would identify a relevant health condition, but also increased the likelihood of underestimating disease severity.
Participants often omitted key details when searching with chatbots or received responses that were difficult to interpret.
Index analysis
Researchers described the findings as indicating a failure in two-way communication between users and AI systems.
Participants using chatbots did not make better decisions than those using traditional methods such as online searches or their own judgment.
Governance controls
The study notes that users often do not know what information to provide to chatbots to receive the best possible health advice.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.