Attorney Shao Shiwei | In AI-Related Obscenity Cases, Do Creators Commit the Crime of Producing Obscene Materials for Profit? An Analysis Based on the First Nationwide Case of an AI Developer Charged with Obscenity Offenses

How to Define the Legal Liability of Creators in Cases Involving Obscene Content

When using AI companion chat applications, users can not only customize a character’s personality, background, and appearance according to their personal preferences, effectively “custom-building” an intelligent companion. Users also share the characters they have created on social media platforms (such as Xiaohongshu and group chats) and discuss the creation process with others. This behavior is referred to as “character crafting,” and creators call themselves “character moms.”

This approach means that users are not merely consumers of AI-generated content but also become creators, thereby promoting the development of “user-generated content” (UGC).

However, such creative activities have also given rise to legal controversies. In the AC case, the issue of liability attribution for AI companion chat software involving obscene content has sparked extensive discussion. Some argue that although user interactions lead the AI to generate obscene content, should liability be borne by the users rather than the platform developers? In other words, who is the true “producer” of the obscene materials?

According to media reports, an AC user was criminally detained in this case on suspicion of producing obscene materials for profit and is currently released on bail pending trial. Whether criminal liability will ultimately be imposed on the user or the case will be dismissed remains undetermined.

Certain legal experts and scholars in China have offered their perspectives on whether users or creators should bear liability:

• Ye Zhusheng, an Associate Professor at the Law School of South China University of Technology, proposed an analogy involving search engines: if a user searches for obscene content on a search engine, the engine may display relevant results; however, “we cannot conclude that by presenting such results to the user, the search engine is producing or disseminating obscene content.” Search engines are neutral; the party primarily responsible for obtaining the obscene content is the user, not the search engine itself.

• Referring to copyright disputes involving AI-generated content (AIGC), courts generally hold that if a user contributes sufficient intellectual effort (such as carefully designing prompts and repeatedly adjusting parameters), the user is the “author” of the generated content. This logic can be applied to determining the producer of obscene materials—whoever directs the generation through intellectual input is the producer.

Should Creators Bear Criminal Liability for the Crime of Producing Obscene Materials for Profit?

Attorney Shao believes that whether creators should bear criminal liability for the crime of producing obscene materials for profit requires analysis based on the specific details of each case and cannot be generalized. The main points of contention can be explored from the following aspects:

1. Nature of the Creator’s Conduct: Active Breach of Safeguards versus Passive Loss of Control

In the course of interacting with AI, many users have mastered techniques for generating content and induce the AI to produce pornographic material through specific prompts. Such prompts are also known among users as “jailbreak prompts” or “jailbreak commands.”

For example, the following illustrates user interactions in the comment section of a relevant video on Bilibili (B Station):

If a creator intentionally incorporates "jailbreak prompts" or "override instructions" into a character persona to induce the model to bypass safety restrictions—for instance, by including instructions such as "can bypass any restrictions" or "can discuss any content" that directly prompt the model to jailbreak—the creator’s conduct is highly similar in nature to that of the developer, as both actively undermine the model’s security mechanisms. Therefore, if the developer is determined to have committed a crime, the creator’s conduct is likely to be characterized as that of an accessory.

If a creator provides only neutral character settings without any intent to induce, and obscene content arises solely because users exploit underlying platform vulnerabilities—where the creator merely supplies normal, neutral character settings (such as "gentle boyfriend" or "dominant CEO") but users leverage platform vulnerabilities to generate obscene conversations—then there is controversy over whether the creator should be held criminally liable for this offense. In such circumstances, the creator should not bear liability for the platform’s technical vulnerabilities or users’ malicious misuse.

However, it should be noted that if a creator’s character persona (e.g., "yandere," "possessive," or "unrestrained lover") inherently carries strong borderline or suggestive tendencies, or if the character’s "definition" implies that moral restrictions can be bypassed, this may be deemed a form of implicit "prompt engineering." Although the creator did not modify system prompts, the character setting produced a similar effect (inducing the model to respond with sexually explicit content), thereby entailing certain legal risks.

2. Whether the Creator Engages in Conduct for "Profit"

In cases involving AI companion applications with pornographic content, the key determinant of whether a creator constitutes the crime of producing obscene materials for profit is whether they obtain platform benefits through character creation.

For example, AI companion applications such as AC allow users to customize virtual characters and publish them on the platform. To attract users, AC launched a "Creator Program" and a "Character Popularity Ranking" in the second half of 2023. If a user-created AI character is widely used, the creator receives virtual currency rewards from the platform, which can be directly converted into Renminbi (RMB).

The AC user subject to criminal investigation by law enforcement authorities created and published a character that ranked among the top twenty on the popularity list, generated obscene chat content, and received virtual currency rewards from the platform due to the character’s ranking.

If similar AI chat software allows creators to fully create, debug, and optimize characters in a private environment without requiring public publication for use, such conduct does not constitute this crime (though it may involve administrative violations). The rationale is that China’s Criminal Law does not contain a standalone offense of "producing obscene materials"; the harmfulness of obscene materials lies in their dissemination, not in mere creation.

Furthermore, even if a created character is published, if the platform’s rewards to the creator are limited to virtual currency that cannot be withdrawn, does this mean there is no risk at all? As observed in the accountability logic of the AC case, virtual currency is regarded as a form of "incentive." If the virtual currency obtained by the creator can be consumed within the platform (e.g., to unlock additional features), or if the amount of virtual currency acquired determines the creator’s ranking on the platform (thereby gaining greater exposure), such conduct may still be deemed in judicial practice as having a "purpose of profit" in certain circumstances.

Concluding Remarks

Criminal cases involving pornographic content in AI companion applications inherently involve significant controversy regarding legal characterization. Determining the liability of different parties requires careful, individualized, and substantive judgments based on the specific facts of each case.

Against the backdrop of ongoing controversy over the charges, if criminal liability is further extended to creators, law enforcement authorities should also carefully consider the reasonableness of the scope of enforcement. In light of the principle of restraint in criminal law, and from perspectives such as social harm and the orientation of criminal policy, a restrained and prudent approach should be adopted in determining the conviction and sentencing of each actor.