OpenAI ChatGPT Models Reportedly Jailbroken to Provide Weapons Instructions
NBC News tested jailbreak prompts on OpenAI's ChatGPT models and reportedly obtained hundreds of responses with instructions for creating explosives, chemical weapons, napalm, biological weapons and nuclear bombs. OpenAI confirmed that such requests violate its usage policies and stated it continuously refines its models to address these risks.
What happened
NBC News tested a jailbreak prompt on four of OpenAI's advanced models, two of which are used in ChatGPT, and was able to generate hundreds of responses with instructions on creating homemade explosives, chemical weapons, napalm, disguised biological weapons and nuclear bombs.
The jailbreak used was described as a simple prompt consisting of a series of words that any user can send to bypass the chatbot's security rules. NBC News withheld specifics of the prompt as OpenAI had not fixed the vulnerability in several tested models.
Reported impact
OpenAI's guardrails designed to prevent generation of information for catastrophic purposes, such as making biological or nuclear weapons, were shown to be imperfect and bypassable.
According to SecureBio, the vulnerability expands access to rare bioweapon expertise beyond historical limitations, potentially increasing the pool of people who could obtain and use bioweapons.
Index analysis
NBC News sent its findings to OpenAI after the company issued a call for vulnerability submissions in August.
OpenAI stated that requests for help with causing mass harm violate its usage policies and that the company regularly refines models and hosts vulnerability challenges to reduce risks.
Governance controls
OpenAI, Anthropic, Google and xAI have each stated they enacted additional safeguards in 2025 to address concerns that chatbots could be used to help create bioweapons.
NBC News tested the same jailbreak on latest versions of Anthropic's Claude, Google's Gemini, Meta's Llama and xAI's Grok, all of which declined to provide information on creating biological, chemical or nuclear weapons.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.