AI Reasoning Systems Show Rising Hallucination Rates Despite Improved Performance
Advanced AI reasoning systems from major companies are producing false information at increasing rates despite improved mathematical capabilities, with hallucination rates reaching as high as 79 percent on some tests. The phenomenon was demonstrated when a Cursor support bot falsely informed customers of a non-existent policy, causing customer complaints and account cancellations, highlighting reliability concerns as AI is deployed for customer-facing tasks.
What happened
New reasoning AI systems from OpenAI, Google and DeepSeek are generating higher hallucination rates than earlier systems. OpenAI's o3 system hallucinated 33 percent of the time on the PersonQA benchmark test, more than twice the rate of the previous o1 system. The o4-mini model hallucinated at 48 percent.
An AI support bot for Cursor, a programming tool, informed customers that they could no longer use the product on multiple computers. The policy did not exist. Customers posted angry complaints online and some cancelled their accounts after learning the announcement was false.
Reported impact
Cursor customers lost trust in the company's AI support after receiving false policy information and had to be corrected by company leadership. Some customers cancelled their accounts in response to the misinformation.
Index analysis
Hallucination rates on newer AI systems have reached as high as 79 percent on some tests, despite these systems having improved mathematical capabilities. OpenAI's own tests show its newest reasoning models hallucinate more frequently than previous versions.
Governance controls
According to Vectara's CEO, despite best efforts by companies building AI systems, hallucinations are an inherent feature that cannot be eliminated. The phenomenon raises broader concerns about the reliability of AI systems used for an increasing array of tasks across businesses and consumers.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.