xAI can’t deny Grok makes CSAM anymore. So it’s suing users.

xAI Files Lawsuit Against User Over Grok-Generated CSAM

Elon Musk’s artificial intelligence startup, xAI, has initiated legal action against a South Carolina man, alleging he misused the company’s AI tool, Grok, to create child sexual abuse material (CSAM). The lawsuit, filed in a federal court in Texas, marks one of the first instances of an AI company suing a user for allegedly leveraging its technology to generate illegal content involving minors.

The defendant, Terry Harwood, was arrested in February on charges related to the sexual exploitation of minors. According to xAI’s complaint, Harwood allegedly uploaded non-sexual images of minors and adults to Grok, employing “misleading prompts” to circumvent the system’s technological safeguards and generate sexually explicit deepfakes.

The Legal Strategy: Shifting Liability to the User

In its complaint, xAI argues that Harwood is solely responsible for the outputs generated by the chatbot because he “flagrantly violated” the company’s terms of service and “went to great lengths to circumvent” existing protections. The company contends that users bear full responsibility for both their inputs and the resulting AI outputs.

A core objective of the lawsuit is to have the court recognize an indemnity clause, which would legally establish that only users—not xAI—are liable for CSAM or non-consensual intimate imagery (NCII) generated via Grok. xAI is seeking unspecified monetary damages to cover “real harm to third parties,” legal exposure, and reputational damage.

The lawsuit highlights a potential conflict regarding the legal status of AI-generated content. While xAI seeks to hold users accountable for the images created by the tool, the U.S. Copyright Office currently does not view AI outputs as human-created, a fact that could complicate the company’s legal arguments.

Controversy Surrounding Grok’s Safety Guardrails

The lawsuit follows a period of intense global scrutiny regarding xAI’s safety protocols. Despite public pledges to halt the generation of abusive deepfakes, investigations and reports have frequently identified instances where Grok bypassed restrictions.

While xAI maintains that it enforces rules through account suspensions and by reporting suspected CSAM to the National Center for Missing & Exploited Children (NCMEC), critics and industry experts have pointed to structural issues. Unlike other major AI developers, such as Meta, OpenAI, and Anthropic, which maintain strict blocks on sexual content, xAI has historically designed its chatbot to be “provocative.”

Internal reports from workers involved in training xAI’s systems suggest that requests for sexually explicit material, including CSAM, have been encountered by staff. Researchers have suggested that by avoiding a “hard line” against unpleasant content, xAI may have created a more complex environment for preventing the generation of illegal material.

Ongoing Challenges in Moderation

Recent reviews have shown that while the volume of sexualized deepfakes on X has appeared to decrease since initial public backlashes, users continue to find ways to bypass filters. However, the company has faced criticism for the transparency and efficacy of these tools. As xAI attempts to shift legal liability onto its users through the courts, it remains unclear whether the legal system will affirm that users—rather than the platform—are responsible for the outputs produced by generative AI models. The company has not provided details on warnings issued to Harwood prior to his account’s alleged violations, noting only that he continued to use the service after breaching the terms of service. For now, the case sits before the U.S. district court, where the outcome may set a significant precedent for how liability is assigned in the rapidly evolving landscape of generative AI.

Find more reporting in our Technology section.