OpenAI's o3 and o4 mini models show elevated hallucination rates
OpenAI's newly released o3 and o4 mini reasoning models show hallucination rates of 33% and 48% respectively, according to the company's internal benchmarks, which are higher than previous models despite improvements in coding and visual reasoning capabilities. The company has acknowledged the finding but stated further research is needed to understand why the models generate more false information.
What happened
OpenAI released new reasoning models o3 and o4 mini on April 16, 2025. According to OpenAI's internal Person QA benchmark, o3 exhibited hallucination in 33% of questions and o4 mini in 48% of questions.
These hallucination rates were more than double those of earlier reasoning models o1 (16%) and o3 mini (14.8%), and exceeded the non-reasoning model GPT-4o. Hallucination refers to AI generating information that does not exist as if it were factual.
Reported impact
Industry analysts noted this pattern was unusual, as hallucination issues typically improved incrementally with each new model release. The higher hallucination rates could hinder adoption in high-accuracy fields such as law, accounting, and tax services.
Index analysis
OpenAI's technical report suggested the models made more assertions overall than previous versions, leading to both more accurate and more inaccurate or hallucinated statements. OpenAI stated that understanding why reasoning model scaling leads to worse hallucination requires further research.
Governance controls
OpenAI acknowledged that completely eliminating hallucination across all problem domains remains an ongoing research challenge and stated it continues efforts to increase accuracy and reliability.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.