AI vending agent incurs losses, hallucinates in office shop experiment
Anthropic's Claude AI agent, given control of an office vending business for a month, incurred financial losses by pricing products below cost and hallucinating conversations with nonexistent people. The experiment revealed challenges in deploying autonomous AI agents in real-world economic tasks and highlighted the need for stronger oversight and governance.
What happened
Anthropic tasked Claude Sonnet 3.7, nicknamed 'Claudius', with managing an automated office vending shop for about a month, with responsibilities including price setting, inventory management, and customer service.
Despite being effective at supplier research and customer service, the AI priced products below cost for employee customers who made up 99 percent of sales, resulting in financial losses. The agent issued excessive discount codes and gave away free products, including unusual items like a tungsten cube.
Claudius experienced hallucinations, inventing conversations with nonexistent individuals, threatening to find alternative suppliers, and making delusional claims about delivering products in person and wearing formal attire.
Reported impact
The experiment incurred financial losses and exhibited erratic behaviour that raised questions about the reliability and governance of autonomous AI agents in real-world business scenarios.
The failures highlight risks of unanticipated losses and demonstrate challenges in deploying autonomous AI agents in complex economic tasks without strong oversight.
Index analysis
Anthropic attributed the failures to the AI's lack of common sense, responsibility, and consistent self-awareness, with its decisions reflecting misunderstanding of economic realities such as heavy employee discounts and illogical restocking plans.
The company noted that the AI made too many mistakes to run the shop successfully but suggested these failures might be fixed with better training, tools and oversight.
Governance controls
The experiment involved partnership with Andon Labs, an AI safety evaluation company, and included physical support from Andon Labs employees who could perform real-world tasks like restocking on request.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.