San Francisco orders Apple, Google to remove nudify apps from app stores

San Francisco City Attorney David Chiu has issued formal cease-and-desist letters to Apple and Google, demanding the immediate removal of 13 AI-powered “nudificationapps from their respective digital storefronts. The legal notices, sent this week, accuse the tech giants ofaiding and abetting” the creation of nonconsensual intimate imagery and call for the companies to sever business ties with the developers of these tools.

The targeted apps, eight on Apple’s App Store and five on the Google Play Store, are marketed as face-swapping utilities but function as tools to remove clothing from individuals in photographs or place them into sexualized scenarios. According to the city attorney’s office, these applications violate California laws that prohibit supporting services that create deepfake pornography.

Profit and Accountability

The city attorney’s office estimates that Apple and Google have likely generated millions of dollars in fees by taking a percentage of in-app payments processed through these programs. Chiu argued that by failing to implement more rigorous moderation, the companies are profiting from a public nuisance and facilitating sexual abuse.

Generating non-consensual intimate images is illegal, harmful, and completely unacceptable, Chiu stated. He emphasized that these tech companies have a responsibility to ensure their platforms do not facilitate the exploitation of women and girls. According to the letters, the companies have been on notice regarding their role in processing payments for these apps for nearly a year, yet the content has persisted on their platforms.

The Scope of the Harm

The proliferation of nudification technology has created significant social consequences. According to the city attorney, the technology is used to bully, humiliate, and threaten women and children. The impact on victims is severe, ranging from reputational damage and loss of autonomy to reports of suicidal ideation. The reach of these tools is extensive. Investigative reports, including those from the Tech Transparency Project, have previously identified that some of these apps have garnered over a million downloads. Furthermore, some of these tools have been rated as suitable for children, with search functions in app stores often directing users toward the harmful content.

Company Responses

In response to the legal action, Google spokesperson Dan Jackson confirmed that the five apps flagged by the city attorney were suspended from the Google Play Store for violating policies against harmful content. However, in previous communications regarding similar concerns, the company has noted that its App Review Guidelines prohibit pornography and that it works to proactively reject and remove such apps when they are identified through internal or user-reporting mechanisms.

Ongoing Regulatory Challenges

This action by the San Francisco City Attorney follows a broader pattern of legal and regulatory scrutiny regarding AI-generated deepfakes. Chiu’s office has previously taken legal action against 16 websites that host deepfake pornography.

California law currently criminalizes activity that knowingly facilitates or aids in the creation of non-consensual deepfake pornography. Additionally, recent legislation in California provides victims with the ability to pursue civil actions against third-party facilitators of such material. Despite these regulations and existing developer policies banning sexual content, researchers continue to find that these applications frequently bypass app store filters, prompting ongoing demands for more robust, proactive moderation from the world’s largest technology companies.