AI Chatbots Fail Majority of Personal Finance Questions in Study
A study by UK fintech firm Saturn tested over 10,000 answers from 18 AI chatbot models on personal finance questions and found failure rates of 57% overall and 88% on complex queries. Despite these high error rates, consumer use of chatbots for financial guidance has grown significantly, with many users reportedly trusting the tools without independent verification.
What happened
Saturn, a UK fintech firm, conducted a study testing 121 financial questions through 18 free and paid AI chatbot models from providers including ChatGPT, Gemini, Claude, and Copilot, with each question repeated up to 5 times to produce over 10,000 answers.
The study scored answers as failures when they contained factual errors, omitted material information, or lacked required warnings. Free models failed on 63% of answers compared to 49% for paid versions, while the hardest, multi-step questions produced failure rates of 88% overall and 93% for free models.
Reported impact
Saturn CEO Amal Jolly stated: 'Millions of people are trusting the AI models for money advice, but they are getting wrong answers that can lose them money.' One pension tax answer reportedly could have exposed a saver to a Β£17,500 charge from HM Revenue and Customs.
A PensionBee survey of 1,000 US adults found nearly six in ten would act on money guidance without independently checking it, and nearly one in four said a chatbot had already given them wrong information about their finances. A global EY survey found 49% of 18,000 consumers had used AI for savings and investment decisions.
Index analysis
Errors in the study included miscalculations, overlooked tax changes, and nonexistent rules. Claude Opus 5 in reasoning mode led the field but still failed 39% of answers.
Research found that 44% of consumers surveyed wrongly believe AI-generated financial information is regulated, and 56% said they trust chatbot financial tools ahead of television and radio at 47%.
Governance controls
Amal Jolly stated that AI financial advice is unregulated, leaving consumers without the compensation rights a human adviser would carry, and urged the FCA to act quickly.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.