Google expands Android content scanning with SafetyCore and ContentSafetyManager
Google expanded its SafetyCore content scanning system on Android devices and introduced ContentSafetyManager API allowing authorized apps to analyze sensitive content on-device, raising privacy concerns. Simultaneously, Google tightened oversight of AI applications generating non-consensual intimate deepfakes, after analysis found such apps on Google Play lacked consent verification mechanisms.
What happened
Google expanded Android System SafetyCore to phones and prepared ContentSafetyManager API, allowing authorized applications to scan sensitive content directly on devices. SafetyCore was distributed through system services in November 2024 and now has over one billion downloads via Google Play.
Users found SafetyCore appeared on their phones without manual installation and struggled to remove it; the service reappeared after manual deletion, and some users sideloaded a placeholder app to prevent reinstallation. Google separately tightened oversight of AI applications that create sexual deepfakes from photos without consent, issuing new guidance prohibiting generation, facilitation and distribution of sexual content created without the consent of the depicted person.
Reported impact
Users expressed distrust of SafetyCore, with posts on forums like r/degoogle reporting unexpected reappearance of the service after removal. Applications like 'AI Video Generator' and 'Kiss AI' available on Google Play lacked mechanisms to verify consent before using photos to generate intimate videos, according to testing by ThePrint.
Some AI deepfake applications on Google Play had millions of downloads despite the policies already prohibiting such content, indicating enforcement gaps.
Index analysis
SafetyCore classification is triggered only when requested by an application using the service, such as warnings of sensitive content in Google Messages, according to Google documentation. Google stated that images and classification results are not sent to its servers for this function.
Google tightened guidance on AI applications after ThePrint analysis found numerous 'kiss and hug' applications that exceeded one million downloads and lacked mechanisms to verify consent before generating intimate content from uploaded photos.
Governance controls
Google updated developer guidelines to clarify that applications generating, facilitating or distributing sexual content created without consent of the depicted person are prohibited on Google Play. The company stated that sensitive content classification in SafetyCore does not send data to Google servers by default, though the service has full network access.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.