OpenAI finds GPT-o3 and o4-mini hallucinate in significant proportion of tests
OpenAI reported that its newest reasoning models GPT-o3 and GPT-o4-mini exhibit substantial hallucination rates in testing, with rates reaching 79% on general knowledge questions. The company suggested the increased fabrication may result from models generating longer and more speculative responses rather than sticking to confident predictions.
What happened
OpenAI's investigation revealed that GPT-o3 hallucinated in 33% of public figure identification tasks, double the rate of its predecessor o1. The smaller model GPT-o4-mini performed worse, hallucinating in 48% of similar tasks.
On the SimpleQA general knowledge test, hallucination rates increased to 51% for GPT-o3 and 79% for GPT-o4-mini. OpenAI stated to The New York Times that the increased hallucinations may stem not solely from complex reasoning but also from models generating lengthier and 'freer' responses.
Reported impact
OpenAI and competitors including Google and Anthropic are concerned that higher hallucination rates contradict their goal of making AI systems useful rather than dangerous. The models' tendency to speculate and improvise creates a blurred line between theoretical reasoning and fabricated facts, with examples cited including false claims such as Abraham Lincoln hosting a podcast.
Index analysis
AI researchers have proposed a theory that as models attempt to reason more deeply, the likelihood of errors increases. Older simpler models stick to high-confidence predictions, while newer reasoning models analyze multiple paths and improvise, which often results in factual invention.
Governance controls
OpenAI and competitors are working to improve accuracy, though no specific control mechanisms were detailed in the report.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.