AI Police Report Software Misinterprets Movie Audio as Officer Transformation in Utah
An AI-powered police report writing tool developed by Axon Enterprise mistakenly generated a report claiming a Utah officer transformed into a frog, after misinterpreting background audio from an animated movie during a domestic disturbance call. The error, which occurred during testing of the Draft One software, was caught and corrected by human review, highlighting reliability concerns with AI systems used in law enforcement.
What happened
The Heber City Police Department tested AI-powered software called Draft One to automatically generate police reports from body camera footage. The software analyzes body-worn camera audio and video to produce written reports using OpenAI's GPT large language models.
During a domestic disturbance call, the software generated a report claiming an officer had transformed into a frog, with skin turning green and slimy. Police Sergeant Rick Keel stated that the AI system mistakenly picked up background audio from the movie 'The Princess and the Frog' playing in the suspect's home and incorporated it into the report narrative.
The Heber City Police Department issued a clarification that no such transformation occurred and the report was corrected through human review.
Reported impact
The incident exposed significant accuracy problems with the AI report-writing tool. Even a simple mock traffic stop demonstration of the tool's capabilities required extensive corrections.
Index analysis
The AI system's failure to distinguish between relevant body camera audio and unrelated background noise demonstrates the technology's difficulty with context and unexpected audio inputs. Experts warned that hallucinations in police reports could escape detection and that automation might reduce officer care in reviewing generated documents.
Critics raised concerns that the technology could perpetuate pre-existing racial and gender biases in law enforcement, and that it could introduce deniability and reduce officer accountability.
Governance controls
The police department confirmed that AI-generated reports undergo human review before finalization. Sergeant Keel stated that this incident 'taught us the importance of correcting these AI-generated reports.'
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.