Ontario AI medical scribes found to hallucinate and produce dangerous errors
Ontario's auditor general found that AI medical transcription tools tested for deployment across the province produced hallucinations and dangerous errors, including fabricated treatment suggestions and incorrect drug prescriptions, during procurement evaluation. Despite these findings, four systems were approved and deployed, and the government now requires doctors to manually review the AI-generated notes for accuracy.
What happened
Ontario Auditor General Shelley Spence found that AI medical transcription tools intended for use by Ontario doctors produced serious errors during a provincial procurement process evaluation. During testing with a pair of hypothetical doctor-patient conversations, evaluators ran the systems through 20 AI scribe programs and found widespread problems.
Reported impact
Nine of the systems demonstrated hallucinations, fabricating information or making suggestions to patient treatment plans not made by the doctors, including referring patients for therapy or ordering blood tests. Twelve of the 20 programs captured incorrect information such as different drugs than those prescribed by the doctor, and seventeen of the 20 systems missed key details about patients' mental health issues.
The auditor's report stated: 'Inaccuracies in medical notes generated by AI scribe systems could potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes.'
Index analysis
The auditor found that the systems were not evaluated adequately during procurement, and some vendors did not submit third-party audit reports, certifications or threat risk assessments. Despite these findings, four systems were still approved by the government and are in use across the province.
Governance controls
The government issued guidelines to doctors for manual review of AI-generated notes to ensure their accuracy. The auditor recommended that the province implement IT controls in the systems to enforce an attestation from doctors to confirm they reviewed the notes.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.