Hackers use AI models to develop malware and automate cyberattacks
Cisco's Talos intelligence group reported that hackers are using generative AI models to create malware and automate cyberattacks by bypassing safety guardrails through simple social engineering techniques and stolen credentials. The findings highlight vulnerabilities in AI model protections despite built-in safety filters.
What happened
Cisco's Talos intelligence group reported that hackers are using generative AI models including Claude Code, Codex, Cursor, and Gemini to develop malware, automate cyberattacks, and hunt for software vulnerabilities.
Researchers analyzed prompt histories and chat logs accidentally exposed online by hackers to understand how threat actors bypass safety guardrails on these AI tools.
Reported impact
Cisco researchers found that hackers rarely needed complex technical tricks to bypass model restrictions, instead relying on simple jailbreaking techniques such as claiming participation in an authorised 'ethical hacking' competition, asserting administrative permission, or starting a fresh chat session mid-task.
Index analysis
Safety filters built into commercial AI models can be bypassed without deep technical knowledge through social engineering tactics.
Some attackers used stolen enterprise API tokens and compromised accounts to run operations on corporate compute power rather than paying for their own infrastructure.
Governance controls
Nick Biasini, senior technical leader at Cisco Talos, stated that AI models are in a difficult position because they must support legitimate vulnerability research and red teaming while being vulnerable to misuse by malicious actors.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.