AI Chatbots Found to Dispense Inaccurate and Potentially Harmful Medical Advice
A study by researchers at Canadian and UK universities found that popular AI chatbots including ChatGPT, Gemini, Grok and Meta AI frequently hallucinate and produce inaccurate or misleading medical information in response to evidence-based health questions, with half or more of answers deemed problematic.
What happened
A study by researchers from the University of Alberta and Loughborough University tested five AI chatbots by posing 50 medical questions on topics including vaccines, cancer, stem cells, nutrition, genetics, exercise and fitness.
The researchers found that half of the answers to evidence-based medical questions were 'somewhat' or 'highly' problematic. Grok returned the most problematic responses (58%), followed by ChatGPT (52%) and Meta AI (50%).
The chatbots often 'hallucinate,' generating incorrect or misleading responses due to biased or incomplete training data. Previous analysis found that only 32% of more than 500 citations from ChatGPT, ScholarGPT and DeepSeek were accurate, with almost half at least partially fabricated.
Reported impact
Experts warned that the incorporation of AI chatbots into medicine requires diligent oversight, noting that chatbots are not licensed to dispense medical advice and may not have access to up-to-date medical knowledge.
Index analysis
The chatbots performed best answering questions about vaccines and cancer, and worst with stem cells, athletic performance and nutrition.
Researchers noted that chatbots do not reason or weigh evidence, nor are they able to make ethical or value-based judgments.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.