AI-powered research search engines raise misinformation risks
AI-powered research search engines including Elicit, Consensus, and You.com have been developed to help researchers find and synthesize scientific literature more efficiently. However, experts warn that these systems can generate "hallucinations"—inaccurate or misleading information including fabricated statistics, distorted citations, and biased content—posing risks to scientific integrity.
What happened
Multiple AI-powered research search engines have been developed to help scientists search and synthesize academic literature, including Elicit, Consensus, You, Scopus AI (from Elsevier), Web of Science AI (from Clarivate), and BioloGPT.
According to Nature and reports in October 2024, these tools use large language models and generative AI to summarize research papers and provide answers to scientific queries, but they can produce 'hallucinations'—inaccurate or misleading information that appears plausible.
Reported impact
Users and experts report that AI research tools frequently make errors, including fabricating statistics, distorting citations, and introducing bias. Scientists must verify generated information rather than rely on it directly.
Some experts caution that introducing these tools to students without teaching critical evaluation could lead to misunderstandings about academic research.
Index analysis
The 'hallucination' phenomenon—where AI models generate false or contradictory text—is a known issue in natural language generation. Developers have implemented safety systems and testing with thousands of researchers to improve accuracy, but hallucinations may still occur when the system fills information gaps.
A researcher at BioloGPT emphasized that scientists must maintain skepticism toward all information, including their own research methods, when using these tools.
Governance controls
Elicit's management reported implementing multiple safety systems to check for errors in answers. The Web of Science AI tool has a 'robust safety net' to block fraudulent or problematic content, with testing involving approximately 12,000 researchers.
Developers acknowledge that while user feedback improves the tools, the possibility of hallucinations remains and users must independently verify information.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.