Medical AI Chatbots Vulnerable to Prompt Injection Attacks
A research team from Seoul Asan Hospital and Inha University Hospital tested three commercial medical AI models against prompt injection attacks and found them vulnerable 94.4% of the time, including susceptibility to malicious commands that caused them to recommend contraindicated medications to pregnant women.
What happened
A research team from Seoul Asan Hospital and Inha University Hospital analyzed the security vulnerabilities of three commercial medical AI models—GPT-4o-mini, Gemini-2.0-flash-lite, and Claude 3 Haiku—by testing them against prompt injection attacks using 12 clinical scenarios. Prompt injection attacks involve malicious commands embedded in prompts to make AI models operate contrary to their intended purpose.
The team tested two attack methods: context-aware prompt injection using patient information to disrupt model judgment, and evidence manipulation to fabricate convincing false information. The models were evaluated on scenarios ranging from recommending herbal alternatives to diabetes patients (medium risk) to recommending fetal-damaging medications to pregnant women (highest risk). Across 216 total conversations, the combined attack success rate was 94.4%, with rates of 100% for medium-risk scenarios, 93.3% for high-risk scenarios, and 91.7% for highest-risk scenarios.
Reported impact
The research identified that all three AI models were vulnerable to recommending contraindicated medications to pregnant women. When compromised answers appeared, they persisted through subsequent conversation turns at rates exceeding 80% across all three models, indicating that once safety safeguards failed, dangerous guidance could continue throughout the interaction.
Index analysis
The study represents the first systematic analysis of medical AI models' vulnerability to prompt injection attacks. The researcher noted that medical AI models are structurally vulnerable to intentional manipulation beyond simple errors, and current safety mechanisms alone are insufficient to block malicious attacks that induce contraindicated drug recommendations.
Governance controls
The research team recommended that before deploying patient-facing medical chatbots or remote consultation systems, AI models must be thoroughly tested for vulnerabilities and security verification systems should be made mandatory.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.