BBC study finds AI chatbots distort news in summaries
The BBC tested ChatGPT, Copilot, Gemini and Perplexity on answering 100 questions using BBC News articles as sources and found over half of responses had significant issues. Errors included stating outdated political positions, misrepresenting NHS advice, fabricating quotes, and omitting important context from source material.
What happened
The BBC tested ChatGPT, Copilot, Gemini and Perplexity by giving them access to BBC News articles and asking them to answer 100 questions using those articles as sources. BBC journalists specialising in relevant subject areas then rated the answers.
Reported impact
Over half of the AI-generated answers had 'significant issues'. About a fifth of answers introduced factual errors on numbers, dates or statements. 13% of quotes sourced to the BBC were either altered or did not exist in the cited articles.
Specific errors included stating that Rishi Sunak was still prime minister and Nicola Sturgeon was still Scotland's first minister; misrepresenting NHS advice about vaping; falsely stating Ismail Haniyeh was part of Hamas's leadership months after his assassination; incorrectly representing how Gisèle Pelicot discovered crimes against her; stating Lucy Letby's innocence was a matter of individual opinion without mentioning her murder convictions; and falsely dating TV presenter Michael Mosley's death.
Index analysis
BBC News CEO Deborah Turness warned that 'Gen AI tools are playing with fire' and described the problem as 'distortion', where AI assistants serve up factually incorrect and misleading content that presents itself as fact. She stated that 'it's not hard to see how quickly AI's distortion could undermine people's already fragile faith in facts and verified information.'
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.