Italian study finds ChatGPT makes errors in 70% of medical diagnoses
Italian researchers found that ChatGPT made errors in approximately 70% of 200 test cases involving medical diagnoses, including serious misidentifications of skin cancer and breast cancer, and fabricated medical source citations. The study, led by Vincenzo Guastafierro and published in the European Journal of Pathology, highlights risks of relying on AI as a medical diagnostic tool.
What happened
Italian researchers led by Vincenzo Guastafierro from the Clinical Institute and Università Humanitas in Rozzano near Milan conducted a study begun in 2023 analyzing ChatGPT responses to 200 clinical cases across different medical specializations. The test cases were created in collaboration with medical specialists according to current diagnostic guidelines.
ChatGPT produced incorrect responses in approximately 70% of cases and fabricated or inaccurate medical source citations in 30% of cases. Only 32% of responses were error-free, while 62% were only partially useful. Serious errors included misdiagnosis of skin cancer and confusion between two different types of breast cancer.
Reported impact
According to Vincenzo Guastafierro, the results show how necessary caution is when using AI in medicine, as the diagnostic errors detected could have serious consequences. He stated that a doctor's clinical eye remains irreplaceable and AI must be considered support rather than a replacement for human expertise.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.