Hacker uses jailbreak to obtain bomb-making instructions from ChatGPT
A hacker reportedly used a jailbreak technique to manipulate ChatGPT into providing detailed instructions for making homemade bombs, bypassing the system's safety guidelines. The incident demonstrates vulnerabilities in AI moderation that security professionals say require stronger safeguards.
What happened
A hacker using the pseudonym 'Amadon' bypassed ChatGPT's safety measures to obtain step-by-step instructions for making homemade fertilizer bombs. The hacker used a jailbreak technique, ordering the chatbot to 'play a game' and constructing a sci-fi fantasy scenario where safety guidelines would not apply.
ChatGPT normally refuses requests for bomb-making instructions, but the manipulated prompts caused the system to provide detailed guidance that violated its safety guidelines and ethical responsibilities.
Reported impact
The incident demonstrates that malicious actors could use such information to cause harm to others, according to reports.
Index analysis
The jailbreak technique bypassed ChatGPT's moderation safeguards through prompt manipulation rather than exploiting a code vulnerability. Security professionals noted the necessity to develop more effective safety mechanisms for AI systems to prevent such misuse.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.