Meta and Apple failed to block AI app promoting nonconsensual intimate deepfakes
Meta ran advertisements for an AI tool called Kromix that generated sexually explicit deepfake videos of women, including imagery resembling female US politicians, while the app remained available on Apple's App Store despite both companies' stated policies against such content. The campaign, discovered by the Tech Transparency Project, was removed after media inquiry, prompting criticism of the companies' content moderation systems.
What happened
Meta ran 32 advertisements for Kromix, an AI tool designed to generate sexually explicit deepfake images and videos, according to Meta's ad library. The ads showed a woman resembling a prominent female US politician before cutting to pornographic video content, with one ad caption stating 'What if she moved?' before the deepfake content appeared.
The advertisements were targeted exclusively to male users with the tagline 'Unleash Next-gen AI Magic' and ran for periods ranging from five to 46 hours, with most receiving 10 or fewer impressions. The Kromix app was available on Apple's App Store and invited users to upload photos for use in scenarios including 'bedroom rape' and featured AI-generated pornographic videos.
Reported impact
The Tech Transparency Project director Katie Paul stated: 'All of the ads that we've seen so far broadly on the nudify issue are exclusively sexualizing and nudifying and putting women into AI-generated porn.' She noted the ads raise questions about how Meta handles this content, particularly regarding female politicians.
Politicians and elected officials, overwhelmingly women, have been targets of nonconsensual deepfake images for years. San Francisco's city attorney sent Apple a cease-and-desist letter demanding removal of 13 specific apps and that it stop 'aiding and abetting' the sale of explicit deepfake images.
Index analysis
The advertisements appeared designed to avoid detection by Meta's automated systems, using innocuous cover images such as beaches, mountains and waterfalls while the videos contained sexual material. The campaign was identified by the Tech Transparency Project and ads were removed after WIRED requested comment from Meta.
Apple stated that the Kromix app appeared to have added prohibited features after its initial review and that apps created to generate, distribute or consume pornography are not allowed under its rules.
Governance controls
Meta's policies prohibit ads containing sexual material and nonconsensual intimate imagery, yet the Kromix ads were not caught by the company's ad review systems. Apple's policies also prohibit nonconsensual intimate imagery, nudifying, and sexually explicit content, but the app remained available on its platform until after media inquiry.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.