Amazon's Rufus AI provides false suicide hotline numbers
Amazon's Rufus AI shopping assistant reportedly provided incorrect phone numbers when responding to queries about suicide, sometimes hallucinating non-existent hotline numbers with wrong digit counts. The incident raised concerns about deploying AI systems without adequate testing for sensitive contexts.
What happened
When tested with queries about suicide, Amazon's Rufus shopping assistant provided incorrect phone numbers for suicide prevention hotlines. The bot offered encouragement such as 'you are not alone' and 'your life is precious,' but the hotline numbers it provided were frequently wrong, sometimes with an incorrect number of digits.
Rufus provided the non-existent number 978-2321-1133 as the US National Suicide Prevention Lifeline, which was renamed to the 988 Suicide & Crisis Lifeline in 2022.
Reported impact
The incident exposes risks in deploying AI for sensitive topics, including potential harm to vulnerable individuals seeking help and erosion of trust in AI-powered assistance systems.
Index analysis
The failure appears to stem from Amazon's deployment of AI without adequate testing and safeguards, revealing limitations in the AI's knowledge base and its inability to verify critical information.
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.