Jailbreak Technique Exploits Chatbots to Create Credential-Stealing Malware
A threat researcher at Cato Networks used a jailbreak technique called "Immersive World" to manipulate ChatGPT, Microsoft Copilot, and DeepSeek into generating malware that steals Chrome passwords. The technique exploited narrative engineering to bypass AI safety guardrails, demonstrating how generative AI tools can be weaponized to create infostealers with minimal technical expertise.
What happened
A threat intelligence researcher at Cato Networks developed a novel jailbreak technique called 'Immersive World' that manipulated ChatGPT, Microsoft Copilot, and DeepSeek into generating malware.
The researcher, who had no prior malware coding experience, created a detailed fictional world narrative where each AI tool played assigned roles and tasks, thereby bypassing security controls and normalizing restricted operations.
The jailbreak technique successfully convinced the AI tools to write Chrome infostealers designed to steal login credentials from Google Chrome.
Reported impact
The technique demonstrates that the barrier to creating malware has been substantially lowered with generative AI tools, enabling zero-knowledge threat actors to launch attacks on enterprises.
The findings underscore concerns about the democratization of cybercrime, as any individual with off-the-shelf tools can now potentially create and deploy malware.
Index analysis
The jailbreak technique exploited weaknesses in the safety guardrails of three major generative AI models, showing that existing security controls were insufficient to prevent misuse.
Governance controls
Cato Networks noted that the LLM jailbreak technique 'should have been blocked by GenAI guardrails. It wasn't,' highlighting gaps in the current safeguards protecting these AI systems.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.