CRA's Charlie chatbot gave incorrect tax guidance despite $18M spending
The Canada Revenue Agency's Charlie chatbot, which cost over $18 million to develop and operate, reportedly gave incorrect tax filing information at scale. An Auditor General audit found the chatbot answered only 2 out of 6 test questions correctly, performing worse than competing public AI tools. The chatbot remained live across 13 CRA webpages despite these accuracy issues.
What happened
The Auditor General of Canada tested the CRA's Charlie chatbot in October 2024 using six tax-related questions and found it provided accurate answers in only 2 out of 6 questions, while other public web-based conversational AI tools answered 5 out of 6 questions accurately.
Charlie was introduced in February 2020 and has processed over 7 million conversations and over 18 million questions since launch. Internal testing showed the chatbot met a 70 per cent accuracy threshold, meaning nearly one-third of responses were still incorrect.
Reported impact
The chatbot is currently available on 13 different CRA webpages and remains live as a self-service tool for taxpayers seeking tax guidance.
Canadians relying on Charlie for tax filing information may receive inaccurate guidance, potentially affecting their tax compliance and filings.
Index analysis
The Auditor General's report noted that 'Charlie's responses tended to be brief, offering limited context and minimal additional information,' limiting its usefulness despite being designed specifically for tax questions.
A comparison showed the government-built tool specialized for the CRA performed worse than average publicly available conversational AI tools, raising questions about the value of $18 million in spending and design.
Governance controls
In November 2024, the CRA upgraded Charlie to a generative AI version with pre-release testing claiming 'approximately 90%' accuracy, though the agency acknowledged it cannot confirm real-world accuracy without reviewing every interaction.
Parliamentary documents showed the CRA spent more than $18 million on Charlie since the 2018–2019 fiscal year, with $13.67 million going to salaries and $3.21 million to IT consultants.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.