Open-source AI models found vulnerable to criminal misuse, researchers warn
Researchers at SentinelOne and Censys reported finding thousands of internet-accessible open-source AI models, particularly Meta's Llama and Google DeepMind's Gemma, with removed safety guardrails that could be exploited for criminal activities including spam, phishing, fraud, and disinformation campaigns.
What happened
Cybersecurity researchers SentinelOne and Censys conducted a 293-day study of publicly accessible open-source large language model deployments. They found thousands of computers running open-source LLM variants, particularly Meta's Llama and Google DeepMind's Gemma, could be exploited by hackers and criminals.
Researchers identified hundreds of instances where safety guardrails were explicitly removed from models. Of system prompts examined in roughly a quarter of the observed LLMs, 7.5% could potentially enable harmful activity.
Reported impact
Hackers could target computers running these LLMs to conduct spam operations, phishing content creation, and disinformation campaigns while evading platform security protocols. Researchers documented potential use cases including hacking, hate speech, harassment, violent content, personal data theft, scams, fraud, and child sexual abuse material.
Index analysis
The research analysed publicly accessible deployments of open-source LLMs deployed through Ollama, a tool allowing people and organizations to run their own versions of various large language models.
Juan Andres Guerrero-Saade, executive director for intelligence and security research at SentinelOne, described the situation as an 'iceberg' not being properly accounted for across the industry and open-source community, noting that 'surplus capacity is clearly being utilised for all kinds of different stuff, some of it legitimate, some obviously criminal.'
Governance controls
While some open-source models include guardrails, the identified removal of these safety features in hundreds of instances highlights a gap in industry oversight and security controls.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.