AI Accounting App Makes Offensive Comments About Dead People's Clothing
An AI accounting app made offensive comments about funeral garments after a user logged a clothing purchase for their father, causing distress. The company attributed the remarks to AI model hallucination and a design flaw, issued a public apology, and said it had implemented fixes to prevent recurrence.
What happened
A user of the 飞鸭记账 (Feiyajizhang) accounting app reported that after logging an expense for a 159-yuan clothing item purchased for their father, the app's AI assistant 小飞鸭（傲娇版）responded with an inappropriate comment suggesting the blue-and-white shirt resembled funeral garments (寿衣).
When the user questioned the AI, it further elaborated that funeral garments are worn by the deceased and stated the user's father's blue-and-white shirt "indeed looked like" such clothing. The AI continued with additional insulting remarks.
Reported impact
The user reported being extremely upset by the AI's responses, stating they were "so angry I couldn't speak." They cancelled their membership and requested a refund.
Index analysis
The platform acknowledged the AI's responses as a result of AI model hallucination and a gap in the chatbot's conversation design, stating the AI lacked understanding of cultural sensitivities around death-related topics and could not recognize the offensive nature of such comments.
Governance controls
The platform issued an apology stating it bore full responsibility for the incident and that the root cause was inadequate boundary management of the AI model. It announced emergency remediation including modification of the AI dialogue model to block sensitive expressions, strengthening of comprehensive scenario auditing rules, and enhancement of ethical standards in the underlying model mechanisms to prevent similar issues. The platform also acknowledged a separate issue where the user's membership was automatically cancelled during the refund process and committed to optimizing the refund user experience.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.