University of Pennsylvania AI algorithm for end-of-life care declined during pandemic
An algorithm at the University of Pennsylvania Health System designed to prompt oncologists to discuss end-of-life care with cancer patients degraded during the COVID-19 pandemic, according to a 2022 study. The tool failed hundreds of times to generate the needed prompts, potentially resulting in unnecessary chemotherapy for some patients who would have benefited from advance planning discussions.
What happened
A routine tech checkup revealed an algorithm at the University of Pennsylvania Health System designed to predict patient mortality and prompt oncologists to discuss end-of-life preferences decayed during the COVID-19 pandemic, performing 7 percentage points worse at predicting death.
The tool failed hundreds of times to prompt doctors to initiate important conversations about treatment and end-of-life preferences with patients who needed them.
Reported impact
The prompt failures may have resulted in unnecessary chemotherapy for patients who would have benefited from advance planning discussions with their doctors.
Lead researcher Ravi Parikh believes several algorithms designed to enhance medical care weakened during the pandemic, not just the one at Penn Medicine, suggesting a broader problem across institutions that do not routinely monitor algorithm performance.
Index analysis
FDA Commissioner Robert Califf stated that no single U.S. health system is capable of validating an AI algorithm deployed in clinical care, highlighting systemic capacity gaps.
Researchers note that artificial intelligence systems require consistent monitoring and staffing to function properly, and hospitals may lack the resources needed to validate and maintain these tools.
Governance controls
Many institutions are not routinely monitoring the performance of their AI products, according to researchers involved in the study.
The study was published in 2022 and documented the algorithm's degradation during the pandemic period.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.