UTSA Study: AI-Generated Code Package Hallucinations Pose Security Risks
UTSA researchers conducted a comprehensive study on risks posed by AI-generated code, focusing on "package hallucinations" where large language models recommend non-existent third-party software libraries. The study found that developers relying on AI coding assistance could unknowingly run malicious code uploaded to open-source repositories by bad actors exploiting this vulnerability.
What happened
UTSA researchers led by doctoral student Joe Spracklen conducted a comprehensive study on risks of using large language models (LLMs) to develop software, with results accepted for publication at the USENIX Security Symposium 2025.
The study focused on 'package hallucinations,' where LLMs generate or recommend the use of third-party software libraries that do not actually exist. According to the researchers, up to 97% of software developers incorporate generative AI into their workflow, and 30% of code written today is AI-generated.
Reported impact
Package hallucinations create security vulnerabilities: when an LLM recommends a non-existent package, developers who trust and run the code could unknowingly install malicious code uploaded to open-source repositories by bad actors exploiting this behavior.
The vulnerability is described as simple and ubiquitous, requiring only a single everyday command that programmers typically use daily to trigger the issue.
Index analysis
Package hallucinations represent a novel attack vector, as bad actors can upload malicious code disguised as legitimate packages to centralized repositories used by popular programming languages such as PyPI for Python and npm for JavaScript.
The research highlights that LLMs frequently generate insecure code, and the phenomenon occurs through basic operations that developers perform routinely without specialized knowledge or complex circumstances.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.