Malicious LLMs sold on underground marketplaces to enable cybercrime
Academic researchers identified 212 malicious large language models available on underground marketplaces, designed for cybercriminal purposes including phishing, malware development, and zero-day exploitation. The products, termed 'Mallas', were primarily derived from OpenAI, Anthropic, and open-source models, and researchers documented methods attackers use to bypass commercial LLM safety guardrails.
What happened
Academics at Indiana University Bloomington identified 212 malicious LLMs on underground marketplaces from April through September 2024. The malicious LLMs, termed 'Mallas' by researchers, included uncensored models built on open-source standards and jailbroken commercial models.
Specific malicious products included DarkGPT, sold for 78 cents per 50 messages; Escape GPT, a subscription service costing $64.98 monthly; WormGPT, available for a flat fee with a calculated financial haul of $28,000 over two months; and WolfGPT, available for $150. Nearly all malicious LLMs examined were capable of generating malware, and 41.5% could produce phishing emails.
The malicious products were primarily built on OpenAI's GPT-3.5 and GPT-4, Pygmalion-13B, Claude Instant and Claude-2-100k, with OpenAI being the most frequently targeted LLM vendor. Both DarkGPT and Escape GPT produced accurate, malicious code that went undetected by antivirus tools about two-thirds of the time.
Reported impact
Researchers found that hackers can use malicious LLMs to write targeted phishing emails at scale at a fraction of the cost, develop malware and automatically scope and exploit zero-days. WolfGPT specifically allowed users to write phishing emails that could evade a majority of spam detectors.
Cybercriminals can input malicious prompts that trick LLMs into overriding their guardrails to generate harmful outputs, and can upload documents with hidden instructions that are executed by connected system components.
Index analysis
Tech giants developing AI models have mechanisms in place to prevent jailbreaking and are working on methods to automate detection of jailbreaking prompts. However, hackers have discovered methods to bypass the guardrails, including Microsoft-documented use of a 'skeleton key' to force OpenAI, Meta, Google and Anthropic's LLMs to respond to illicit requests and reveal harmful information.
Researchers from Robust Intelligence and Yale University identified an automated method for jailbreaking OpenAI, Meta and Google LLMs that does not require specialized knowledge such as model parameters.
Governance controls
The academic researchers called for better safeguards against artificial intelligence misuse to address the robust underground demand for malicious LLMs.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.