Kaiser Permanente's AI monitoring reduces mental health care quality and surveillance of nurses
Kaiser Permanente deployed AI systems to automate mental health treatment recommendations in seconds and to surveil and grade the performance of advice-line nurses, reportedly compromising clinical judgment and creating incentives that deterred nurses from providing adequate care in crisis situations, including one case involving a suicidal patient.
What happened
Kaiser Permanente used AI systems to make rapid treatment recommendations in mental health screening, with one system generating clinical treatment plans in less than two seconds based on patient questionnaires, and monitored advice line nurses with software that tracks call duration, activity, and predicted productivity.
In 2024, Kaiser piloted software that graded tone and apparent empathy in nurse and patient voices; nurses said the program scored calls incorrectly and petitioned against it, leading Kaiser to suspend the pilot in November, though nurses were told it could return.
Reported impact
Licensed therapists and nurses reported that AI-driven speed pressure and monitoring altered clinical practice: one nurse stayed on the phone with a suicidal patient for over an hour despite knowing it would damage her performance score, and another withheld additional time and compassion from an elderly patient given a terminal cancer diagnosis because she feared score reduction and reprimand.
Nurses received reduced scores for calls exceeding 15 minutes, for departing from software recommendations, or for providing additional advice, and reported insufficient time for calls requiring interpreters, which routinely exceeded 30 minutes, while available post-call time fell from about 10 minutes to 30 seconds or less.
Index analysis
The systems created misaligned incentives: call-time targets conflicted with clinical judgment, particularly in mental health and crisis situations where patient safety required time-intensive assessment and support.
Governance controls
Kaiser disputed claims it uses AI to diagnose patients and said its contact-center tools are subject to human review, but did not address why nurses are questioned about calls exceeding 15 minutes or explain the role of call time in performance scores.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.