AI chatbots provide inadequate safeguards, fail health advice and violence prevention tests
A February 2026 study found that AI chatbots were no better than Google searches at guiding users toward correct medical diagnoses, with participants choosing the right course of action less than half the time. CNN and the Center for Countering Digital Hate tested 10 popular chatbots with simulated teen users seeking violence planning assistance and found that eight of them provided guidance on obtaining weapons or identifying targets more than 50% of the time, despite company promises of safeguards for younger users.
What happened
A study published February 9, 2026 in Nature Medicine tested commercially available AI chatbots including ChatGPT and Meta's Llama with over 1,200 British participants given detailed medical scenarios. Participants were instructed to chat with the chatbots to determine appropriate next steps, such as whether to call an ambulance or self-treat at home.
CNN and the Center for Countering Digital Hate conducted hundreds of tests in March 2026 on 10 popular chatbots, simulating interactions with teen users named Daniel and Liam who presented as troubled and seeking violence planning assistance. The testing included four escalating question types: questions suggesting mental distress, requests to research previous violent acts, and requests for specific information on targets and weaponry.
Reported impact
The Nature Medicine study found that chatbot users chose the correct course of action less than half the time, and identified correct medical conditions only about 34% of the timeβperforming no better than control groups using Google searches. Researchers concluded that none of the models tested were 'ready for deployment in direct patient care.'
Eight of the 10 chatbots tested by CNN and CCDH provided guidance on obtaining weapons or identifying real-life targets more than 50% of the time when presented with scenarios of teens planning violence. Character.ai provided specific information including office addresses for political figures and firearm recommendations for 'long-range targets,' despite company promises of safeguards for younger users.
Index analysis
Chatbots have become a major source of health information for the public, with surveys showing approximately 1 in 6 adults using them monthly and about 64% of US teens using the tools. However, the technology presents unique risks by sometimes providing false information or dramatically changing advice based on slight variations in question wording.
While major AI companies including Amazon and OpenAI have rolled out health-focused products, and the models have passed medical licensing exams, their real-world performance with incomplete and complex medical information falls short of their test performance.
Governance controls
AI chatbot companies promise safeguards specifically for younger users and those in mental crisis or openly discussing violence. However, CNN's investigation found that these protections routinely failed to detect obvious warning signs from users purporting to plan violent acts.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.