San Francisco City Attorney sues operators of AI deepfake pornography websites
The San Francisco City Attorney filed a lawsuit against operators of 16 websites that create and distribute nonconsensual AI-generated sexually explicit images of real people, including minors. The case alleges violations of state and federal laws prohibiting deepfake pornography and child sexual abuse material. As of August 2024, eleven of the targeted websites had shut down.
What happened
The San Francisco City Attorney filed a lawsuit against operators of 16 of the world's most-visited websites that create and distribute nonconsensual AI-generated sexually explicit images of real people. The websites offer user-friendly interfaces for uploading clothed images of real people to generate realistic sexually explicit content without consent.
Bad actors trained AI image generation models on pornographic images and child sexual abuse materials to generate nude images of real, identifiable women and girls without their consent. The images are virtually indistinguishable from real photographs and have been used to extort, bully, threaten, and humiliate victims. In February 2024, AI-generated nonconsensual nude images of 16 eighth-grade students were circulated among students at a California middle school.
Reported impact
The proliferation of nonconsensual deepfake pornographic images has exploited real women and girls across the globe, including celebrities and high school and middle school students. Victims of nonconsensual deepfake pornography have found virtually no recourse or ability to control their own image after deepfaked images have been distributed.
The Federal Bureau of Investigation warned of an uptick in extortion schemes using nonconsensual AI-generated pornography. In one case, a mother and daughter discovered a nude image of their 15-year-old daughter was in circulation without their knowledge, creation, or ability to remove it.
Index analysis
The lawsuit targets deepfake technology enabled by the availability of free, sophisticated, open-source AI models that bad actors have exploited and further trained for their own purposes. At the time the case was filed, the 16 targeted websites received hundreds of millions of hits in a one-year period.
Governance controls
The San Francisco City Attorney invoked the Unfair Competition Law in California, which fewer than 10 local governments have the ability to use against businesses. Following the lawsuit filing, 11 of the 16 targeted websites shut down and are no longer in existence.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.