TikTok Withheld Algorithmic Safety Feature From Millions of Users
TikTok withheld a safer version of its recommendation algorithm from approximately 15 million US users as a control group in an experiment to test whether the safety feature would reduce engagement. A 16-year-old included in the control group was exposed to thousands of videos about harmful topics before his death by suicide in 2022. TikTok's internal review found the safety protections did not take effect for this user by design.
What happened
In 2021, TikTok developed a safer version of its algorithm designed to stop users from being overwhelmed with harmful content by disrupting repetitive recommendations on topics such as extreme dieting, sadness, and breakups. Instead of rolling out the change to all users, TikTok conducted an experiment: approximately 10% of US users, roughly 15 million people, kept the old version of the app as a control group to measure whether the safety feature would reduce engagement metrics.
Chase Nasca, a 16-year-old from Long Island, was among the 10% kept on the older algorithm. TikTok's algorithm pushed him thousands of videos about suicide, sadness, hopelessness and loneliness. He died by suicide in February 2022. TikTok did not review his account history until more than a year later, after Bloomberg contacted the company in March 2023.
Reported impact
According to TikTok's internal review of Nasca's account, he had become trapped in a filter bubble filled with videos about suicide, self-harm, depression, loneliness, and hopelessness. Because he was part of the control group, TikTok's strategies for disrupting those repetitive recommendations 'did not take effect on this user by design.'
Index analysis
TikTok described the experiment as a 'delicate balance' between user safety and its ability to measure the effect of the safety changes on engagement metrics. The company told Bloomberg that the experiment was intended to confirm whether its safety changes were working. TikTok stated it has since refined its testing practices, although it did not explain what those changes were.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.