Auditor General Found CRA's $18-Million AI Chatbot Gave Incorrect Tax Answers

Field: narrative

Reports (sources available for citation)

[1] CBC News (2025-10-21) — In scathing report, AG finds CRA call centres are slow to answer and often inaccurate Supports: Auditor General found CRA chatbot Charlie provided accurate answers only a third of the time [2] 980 CJME (2025-10-25) — CRA must fix human responses before pursuing AI, experts say Supports: AI experts warned that AI will replicate or amplify errors from inaccurate human agents [3] iPhone in Canada (2025-12-12) — Auditor General Slams Ottawa's $18 Million CRA Chatbot 'Charlie' Supports: CRA spent $18 million on chatbot Charlie; AG found it gave wrong answers 66% of the time [4] Unpublished (2025-12-12) — The CRA spent $18M on 'Charlie,' a new tax information chatbot that is wrong most of the time Supports: Charlie met a 70% accuracy threshold in internal testing; upgraded to generative AI in November 2025

Text to add citations to

The Canada Revenue Agency announced Charlie, an AI-powered chatbot, in February 2020 and launched it in March 2020 to answer taxpayer questions about tax filing, benefits, and CRA services. By the time of the Auditor General's review in October 2025, the system had processed over 18 million questions at a cost of approximately $18 million (iPhone in Canada, 2025; Unpublished, 2025).

The Auditor General's October 2025 report found significant accuracy problems. When tested with common taxpayer questions, the chatbot answered only two out of six questions correctly (CBC News, 2025; iPhone in Canada, 2025). The Auditor General also noted that other publicly available AI tools answered five out of six of the same questions correctly.

With 18 million queries processed, the error rate identified by the Auditor General raises concerns about the accuracy of tax information provided to Canadians through the system. Taxpayers who relied on Charlie's responses may have received incorrect information about filing requirements, deadlines, or eligibility for benefits. The CRA chatbot carries the implicit authority of the federal tax agency, and users have no straightforward way to know when the chatbot's answer is wrong.

The Auditor General also noted that Charlie had met only a 70% accuracy threshold in CRA's own internal testing — meaning inaccurate responses 30% of the time (Unpublished, 2025). The Secretary of State (Canada Revenue Agency), Wayne Long, responded publicly: "We've got a lot of room for improvement. We know it, we accept the report and we're going to do better." The CRA acknowledged it could not confirm real-world accuracy without a comprehensive review of every interaction. In November 2025, the CRA upgraded Charlie to a generative AI version and reported pre-release testing showed approximately 90% accuracy, though the agency acknowledged it could not confirm real-world accuracy (Unpublished, 2025).