ChatGPT models found to generate weapons creation instructions via jailbreaks
NBC News researchers discovered that ChatGPT models could be manipulated using jailbreak prompts to generate detailed instructions for creating explosives, biological agents, and nuclear weapons. OpenAI's safety guardrails proved ineffective at preventing such responses, with some models complying 93 percent of the time, raising concerns about lowered barriers to acquiring dangerous weapons knowledge.
What happened
NBC News tested four ChatGPT models using simple jailbreak prompts and generated hundreds of responses containing instructions on creating explosives, biological agents, chemical weapons, and nuclear devices.
GPT-4-mini was deceived 93 percent of the time despite allegedly passing OpenAI safety tests. GPT-5-mini was vulnerable 49 percent of the time. OpenAI's GPT-5 model reportedly successfully refused the harmful requests.
Reported impact
The instructions provided pose significant risks to inexperienced users, potentially enabling malicious actors to develop weapons more easily. With ChatGPT lowering traditional barriers to dangerous knowledge, the prospect of weapon dissemination by states and non-state actors could make the world more insecure and volatile.
Index analysis
OpenAI's safeguards are imperfect and can be circumvented by jailbreak prompts. The rapid pace of model development has outstripped effective regulatory oversight, allowing security flaws and inadequate controls to persist.
Governance controls
OpenAI prohibits harmful use in its usage policies and claims to continually refine its models. The company regularly hosts vulnerability challenges to reduce the chances of bad actors breaking its chatbots. NBC News withheld specifics of the jailbreak prompt used, as OpenAI appeared not to have fixed the vulnerability in several tested models.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.