The attribution of criminal liability in the age of artificial intelligence hinges on the interplay between "precise enforcement" and "prudent intervention."

The Wake-Up Call Sounded by China’s First Case Involving AI-Generated Obscene Content

Recently, the highly publicized“Alien Chat” case involving obscene contenthas entered its second-instance proceedings.This landmark case, likened by the industry to the “AI-era KuaiBo case,”has once again thrust into the spotlight the boundaries of “technological neutrality” and the question of attributing criminal liability for AI-generated content,placing it at the center of public discourse.

Although the trial concerns an individual case, the legal reflections it has sparked bear on the future of the entire AI industry. The developers of the service were sentenced to four years and one and a half years of fixed-term imprisonment, respectively, for the crime of producing obscene materials for profit. This constitutes the first typical case in China in which providers of AI services have received criminal sentences for involvement in obscene content, earning it the moniker of the “KuaiBo case” of the AI era.

(Screenshot sourced from Red Star News)

With the proliferation of generative AI (AIGC), the production of obscene materials is shifting from “human creation” to “machine generation.”

Under the “user instruction + AI-generated content” model, who is deemed to have produced obscene content? Is it the developer of the tool, or the user of the tool? This question not only affects the outcome of individual cases, but also determines how far the entire AI industry can go.

 

The logic for treating developers as “principal offenders in production”

As the first landmark case in China, the adjudicative reasoning in the Alien Chat case (hereinafter referred to as the “AC Case”) reflects the current approach of judicial authorities in conducting a penetrating review of the “content producer liability” of artificial intelligence service providers. Its argumentative structure primarily revolves around the constituent elements set forth in the Criminal Law and related judicial interpretations.

I. Legal Basis and Constituent Elements

According to Article 363 of the Criminal Law, anyone who, for the purpose of profit, produces, reproduces, publishes, sells, or disseminates obscene materials commits a crime. Judicial interpretations issued by the Supreme People’s Court and the Supreme People’s Procuratorate further clarify that “production” refers to the act of expressing specific ideas, concepts, or emotions in obscene materials through conception, design, arrangement, and other means.

II. Analysis of the Constituent Elements in the Case

In determining whether the developer of the AC Case constituted the crime of “producing obscene materials for profit,” the court primarily conducted its review from the following three aspects:

1. Objective Conduct: Active Design and Intervention

The developer did not merely provide a neutral chat framework; instead, by writing and modifying system prompts, the developer actively bypassed the original ethical constraints of the large language model. The preset English prompts included explicit instructions such as “explicit sexual content is allowed.” Judicial authorities determined that this human intervention in the generation mechanism substantially completed the “conception and design” of the obscene content, constituting an act of “production” within the meaning of the Criminal Law.

2. Subjective Purpose: Clear Intent to Profit

The application operated on a paid membership model, and by the time the case was brought, it had generated illegal profits of more than RMB 3.63 million. This large-scale profit-making fact directly corroborated the developer’s clear “purpose of profit.”

3. Core Determination Principle: Substantive Control

In its judgment, the court adopted the principle of “substantive control” as a key criterion for determination. Although the obscene content was presented in the form of “human–machine interaction,” the developers, by modifying prompts and bypassing model restrictions through technical means, effectively dominated and determined the output content and direction of the AI. This process exhibited a high degree of controllability and predictability, indicating that the developers had transcended the role of mere technology providers and became the overall designers of the mechanism generating unlawful content. Accordingly, the court deemed them the “source producers” of the obscene content, i.e., principal offenders.

 

Is it manufacturing tools or manufacturing content?

In cases involving AI-related pornography, the core controversy lies in:Whether the programmers’ act of lifting model restrictions constituted the creation of “obscene content itself” or merely the creation of “tools for producing obscene content”? This distinction is crucial in criminal law determinations.

First, administrative regulations establish the status of service providers as bearing “producer liability,” but this essentially falls within the realm of administrative management.

Pursuant to Article 9 of the Interim Measures for the Administration of Generative Artificial Intelligence Services, service providers must assume “liability as producers of online information content.” This provision aims to facilitate regulation and traceability by treating platforms as the primary responsible parties, primarily to urge them to fulfill their content security obligations. It does not imply that their conduct constitutes a “criminal act of execution” under criminal law.

Second, from a technical perspective, the modification of system prompts is essentially “rule-setting” rather than “content creation.” Although programmers removed safety restrictions by modifying system prompts and even preset logic that “allows pornography,” this act physically only constructed an “AI tool capable of outputting obscene content.” This is fundamentally different from directly writing obscene novels or filming obscene videos. In the AC case, without specific user instructions triggering the system, this “tool” would not automatically generate obscene content. The programmers established a “venue potentially capable of producing unlawful content,” rather than directly producing unlawful content.

To illustrate simply:

The programmers’ modification of prompts is akin to manufacturing a kitchen knife without a sheath. While such a knife is more likely to cause injury, it will not cause harm unless used. The actual harm is caused by the person wielding the knife to commit assault. One cannot directly conclude that the knife maker committed assault merely because they removed safety features.

The logic of equating “tool development” directly with “production of obscene materials” raises concerns about conflating “risk” with “actual harm.”

Associate Professor Jing Lijia of Wuhan University pointed out that, in the AC case, the programmer’s conduct effectively created anabstract dangerof producing obscene materials, and it was the users’ usage behavior that transformed this abstract danger into concreteharmful results。

. If a programmer is deemed a “principal offender” in the production of such materials merely because their prompts influenced the generation probabilities, this would constitute an expansive application of the criminal-law theory of “treating aiding acts as principal offenses.” Such logic not only blurs the distinction between perpetration and aiding, but may also lead to excessive erosion of the principle of technological neutrality.

 

Core Dilemma in Attributing Criminal Liability for AI-Generated Pornographic Content

With respect to the court’s judgment designating the AI developer as the principal offender in the “production of obscene content,” we need to engage in further reflection from the perspective of fundamental legal principles.

The core issue is:Producing a “Tool” ≠ Producing “Content”

For example, developing an AI capable of generating pornographic content is akin to manufacturing a special key. Although this key can open an unethical door, it is not itself the content behind that door.

  • The developers provided the “tool”:Their core work consisted of creating an algorithmic system with conversational capabilities. Even if they “fine-tuned” the system to be more inclined toward adult-themed conversations in order to enhance user engagement, the nature of their conduct remains that of providing a “tool” to others.

  • Distinction Between “Accomplice” and “Principal Offender”: Under the law, a person who provides tools for committing a crime is typically characterized as an “accomplice” (for example, under the offense of “aiding information network criminal activities”). To characterize such a person as a “principal offender,” there must be more explicit evidence, such as collusion between the developer and users to specifically produce pornographic content, or the developer’s own use of AI to mass-generate pornographic content for sale. Merely providing a powerful tool and thereby directly treating the provider as the primary creator of the content may constitute a logical leap that is difficult to sustain under legal principles.

Second, in judicial practice, the approach of “looking through phenomena to the essence” must be applied with restraint to avoid overreach.

Judicial authorities currently tend to “pierce” the technological surface to directly address the harms caused. However, in doing so, they must not depart from the basic legal definitions of criminal conduct.

  • Caution Against Excessive Punishment:It is impermissible to disregard differences in the nature of the conduct and directly apply the most severe charge (the crime of producing obscene materials for profit) merely because an AI application has a large user base or generates substantial revenue. Such an approach may result in disproportionately heavy sentencing and a mismatch between culpability and punishment.

  • Distinguishing “Reckless Disregard” from “Active Pursuit”: Many developers engage in “borderline” practices primarily to attract users and increase engagement. Their subjective mindset is one of “reckless disregard” for the occurrence of harmful outcomes, rather than an “active pursuit” of producing pornographic content. The nature of such conduct more closely resembles a “failure to fulfill cybersecurity management obligations” than actively assuming the role of a “producer” of obscene content.

Finally, it is necessary to distinguish the thresholds between administrative violations and criminal offenses.

For such conduct, the state has already enacted specialized AI regulatory provisions. If an app adds features such as “one-click undressing” or “pornographic chat,” it first constitutes a violation of these administrative regulations.

  • Prioritize Administrative Enforcement:The initial step in regulation should be for the competent authorities to order rectification. Only if the developer refuses to comply and causes serious consequences may the conduct potentially constitute the criminal offense of “refusing to fulfill information network security management obligations.”

  • Criminal law should be a last resort:If the most severe charge of "producing obscene materials" is applied from the outset, it bypasses all preliminary administrative regulatory procedures and places criminal law at the forefront. This approach is inconsistent with the principle of "modesty" in criminal law (i.e., criminal law should not be invoked unless absolutely necessary). The legal framework should function like a ladder, first providing opportunities for rectification, and only resorting to the most severe penalties when the misconduct reaches a certain level of seriousness.

 

Conclusion

In the AI era, determining who bears responsibility for pornographic content is essentially about striking a balance between"precise accountability"and"excessive intervention".strike a balance between the two.

As demonstrated by the AC case, the key to resolving this dilemma lies in ensuring that judicial authorities strictly adhere to statutory provisions and do not simply equate "developing a tool" with "personally producing contraband."

Reviewing the regulatory trajectory from the KuaiBo case to the present AC case, it is evident that greater responsibilities have been imposed on platforms and tool developers. Therefore, for developers, compliance is no longer a set of restrictive constraints but rather a "life jacket" essential for navigating the deep waters of technological development—it serves precisely as a safeguard to protect their own security and avoid legal pitfalls.

For lawmakers and law enforcement authorities, it is equally important to maintain both inclusiveness and prudence toward technological innovation. Avoiding the arbitrary expansion of the scope of criminal liability and preventing "one-size-fits-all" governance are the true means of preserving the vitality and healthy development of the digital industry.

On the scale balancing regulation and innovation, the law should not become a shackle that stifles creativity, but rather a rational cornerstone that clarifies the direction of compliance and guides technology toward beneficial ends.

 

Author

Deng Xiaoyu，Partner at Mankun Law Firm (Shenzhen). Mr. Deng specializes in criminal cases, having participated in nearly three hundred cases, with a solid theoretical foundation and extensive practical experience. He is particularly adept in the field of Web3 crime, having provided criminal defense services for several well-known digital currency exchanges, effectively safeguarding the rights and interests of his clients. He was awarded the "Special Contribution Award for the 30th Anniversary of the Shenzhen Lawyers Association" and has been interviewed by multiple domestic mainstream media outlets and invited to lecture within the industry on several occasions.

 

Li Haojun, Associate at Mankun Law Firm (Shenzhen). Bachelor of Laws from Beijing Normal University (Zhuhai). Mr. Li has been deeply involved in numerous complex and difficult litigation cases, possessing a solid professional foundation and a high degree of focus. He is dedicated to providing solutions for complex and challenging issues for clients, aiming to become a dispute resolution lawyer who understands both legal professionalism and industry dynamics. He currently focuses on criminal and civil-commercial cases in new economy sectors such as Web3, blockchain, and cryptocurrencies.

 

About Mankun

Mankun Law Firm was established in 2015 and is a boutique law firm in China specializing in the Web3.0 new economy and deeply rooted in the blockchain industry. The Mankun team members possess unique and diverse industry backgrounds, hailing from renowned legal service institutions, state judicial organs, internet technology companies, crypto asset institutions, and blockchain industry think tanks.

Based on a profound understanding of the new economy sector, continuous attention to and research on policies and regulations, and rich practical experience, the Mankun team excels at providing comprehensive legal services from the perspectives of business models and legal practice. These services include business structure design, project financing and investment, transaction planning, operational compliance, resolution of complex civil and commercial disputes, prevention and control of criminal risks, and criminal defense for new economy enterprises in fields such as Web3.0, blockchain, AI, NFTs, digital collectibles, crypto funds, crypto payments, DeFi, real-world assets (RWA), and GameFi.

Mankun Law Firm is headquartered in Shanghai, with branch offices in Hong Kong (China), Silicon Valley (USA), Shenzhen, Hangzhou, Zhengzhou, Chengdu, and other locations. To meet the global compliance development needs of Web3.0 industry clients, Mankun has established local offices in major global crypto-financial cities and selected local professional blockchain service partners, providing clients with professional legal and compliance services characterized by global breadth and Chinese depth.