TikTok algorithm recommended violent content to teenage boys
A former TikTok safety analyst reported that the platform's algorithm systematically recommended more graphic violent and misogynistic content to teenage boys than to girls during his employment from December 2020 to June 2022. He said the system's content moderation relied partly on a 10,000-view threshold before human review and missed 10-15% of harmful content in machine detection.
What happened
Andrew Kaung, who worked as a User Safety Analyst at TikTok's Dublin office from December 2020 to June 2022, conducted an internal investigation and discovered that the platform's algorithm was showing teenage boys more graphic violent and misogynistic content compared to teenage girls.
During the same period at Meta (Instagram's parent company), between 2019 and December 2020, Kaung says the platform relied heavily on users to report harmful videos rather than proactive moderation.
Reported impact
A 16-year-old user named Cai reported being recommended videos featuring violence, someone being hit by a car, misogynistic content, and violent fights after initially seeing benign content like a cute dog video.
Kaung noted that while girls were also shown harmful content, they were more likely to be exposed to content promoting impossible beauty standards, which affected their self-image and confidence.
Index analysis
TikTok uses AI tools to remove the majority of harmful content, but Kaung stated that 10-15% of content can be missed by the machine learning detection model. At the time he worked there, videos that were not removed or flagged by AI were only manually reviewed by human moderators after reaching a threshold of 10,000 views or more.
The recommendation algorithm is fueled by user engagement; if users do not report content, the system interprets this as interest and may recommend more similar harmful content to them.
Governance controls
TikTok states it removes 99% of content violating its rules through AI or human moderators before it reaches 10,000 views, and says it undertakes proactive investigations on videos with fewer views.
TikTok and other major social media companies allow users aged 13 and above to sign up and use content moderation tools combining AI detection and human review.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.