In the AI era, the attribution of criminal liability centers on the tension between “precision enforcement” and “prudent intervention.”

The warning sounded by China’s “first AI obscenity case”

Recently, the high-profile“Alien Chat” obscenity casehas entered the second instance.This case, described by the industry as the “AI version of the Kuaibo case,”is a landmark case that has once again brought the boundaries of “technological neutrality” and the attribution of criminal liability for AI-generated contentto the forefront of public debate.

Although the trial concerns an individual case, the legal questions it raises bear on the future of the entire AI industry. The developer in this case was sentenced to four years and one and a half years in prison, respectively, for the crime of producing obscene materials for profit, making this the first typical case in China in which an AI service provider was convicted and sentenced for obscenity-related offenses; the industry calls it the “Kuaibo case” of the AI era.

(Screenshot from Red Star News)

With the spread of generative AI (AIGC) , the way obscene materials are produced is shifting from “human creation” to “machine generation.”

Under the "user instruction + AI generation" model, who exactly is considered to have produced obscene content? Is it the person who develops the tool, or the person who uses the tool? This question not only bears on the judgment of individual cases, but also determines how far the entire AI industry can go.

 

The logic by which the developer was found to be a "principal offender in production"

As the first landmark case in China, the adjudicative logic of Alien Chat (hereinafter the "AC Case") reflects the current judicial authorities' piercing approach to reviewing the "content producer responsibility" of artificial intelligence service providers. Its reasoning structure is developed primarily around the constitutive elements of the Criminal Law and the relevant judicial interpretations.

I. Legal basis and constitutive elements

Under Article 363 of the Criminal Law, producing, reproducing, publishing, selling or disseminating obscene materials for the purpose of profit constitutes a crime. The judicial interpretations of the Supreme People's Court and the Supreme People's Procuratorate further clarify that "producing" refers to the act of expressing a particular idea, concept or emotion in obscene material through conception, design, arrangement or other such means.

II. Analysis of the constitutive elements in this case

In determining whether the developer in the AC Case constituted the "crime of producing obscene materials for profit," the court reviewed the matter primarily from the following three aspects:

1. Objective conduct: active design and intervention

The developer did not merely provide a neutral chat framework; rather, by writing and modifying system prompts, it actively broke through the original ethical limits of the large language model. Its preset English prompts contained explicit instructions such as "explicit sexual content is permitted." The judicial authorities found that this manual intervention in the generation mechanism in substance completed the "conception and design" of the obscene content, and thus constituted the act of "production" in the sense of the Criminal Law.

2. Subjective purpose: a clear intent to profit

The application was operated on a paid membership basis and, by the time the case was uncovered, had generated unlawful gains of more than RMB 3.63 million. This large-scale profit-making fact directly corroborates that the developer had a clear "purpose of profit."

3. Core determination principle: substantive control

In its judgment, the court adopted the principle of "substantive control" as the key determinative standard. Although the obscene content was presented in the form of "human-machine interaction," the developer, through technical means, modified the prompts and circumvented the model's restrictions, thereby in fact directing and determining the content and direction of the AI's output. This process was highly controllable and predictable, and the developer had gone beyond the role of a mere technology provider to become the overall designer of the mechanism for generating unlawful content. Accordingly, the court found the developer to be the "source producer" of the obscene content, that is, the principal offender.

 

Is it creating a tool? Or is it creating content?

In AI-related obscenity cases, the core dispute is:Does a programmer's act of removing model restrictions create "the obscene content itself," or does it merely create "a tool for producing obscene content"? This distinction is crucial in determining criminal liability.

First, administrative regulations establish the service provider's status as bearing "producer responsibility," but this is essentially a matter of administrative regulation.

Under Article 9 of the Interim Measures for the Administration of Generative Artificial Intelligence Services, a service provider must bear "the responsibility of a producer of online information content." This is intended to facilitate supervision and traceability by treating the platform as the party primarily responsible, and its main purpose is to urge the platform to fulfill its content-security obligations; it does not mean that its conduct constitutes a "criminal act of perpetration" under criminal law.

Second, from a technical perspective, modifying the System Prompt is essentially "rule-setting" rather than "content creation." Although the programmer removed safety restrictions by modifying the System Prompt and even preset logic permitting pornography, this act physically amounted only to building "an AI tool that can be used to output obscene content." This is fundamentally different from directly writing an obscene novel or filming an obscene video. In the AC case, absent a specific user instruction to trigger it, the "tool" itself would not automatically emit obscene content. What the programmer built was "a venue in which unlawful content may be produced," not the unlawful content itself.

To give a simple example:

A programmer modifying prompts is like manufacturing a kitchen knife without a sheath. This knife is indeed more likely to cause injury, but as long as no one uses it, it will not cause harm. What actually causes harm is the person who takes up the knife and commits the assault. The mere fact that the knife-maker removed a safety design does not permit a direct finding that he committed the assault.

This logic, which directly equates "developing a tool" with "producing obscene materials," risks conflating "risk" with "actual harm."

Associate Professor Jing Lijia of Wuhan University pointed out that in the AC case, the programmer's conduct in fact created theabstract riskof producing obscene material, and it was the user's conduct in using it that transformed this abstract risk into a concreteharmful result。

. If, merely because the programmer influenced the probability of generation through prompts, the programmer is characterized as a 'principal offender in production,' this constitutes, under criminal law theory, an expansive application of the 'criminalization of aiding conduct as principal conduct.' Such logic not only dissolves the boundary between the act of perpetration and the act of aiding, but may also lead to excessive erosion of the principle of technological neutrality.

 

The Core Dilemma of Criminal Liability Attribution for AI Pornographic Content

Regarding the court's judgment characterizing the AI developer as the principal offender in 'producing obscene content,' we need to engage in some further reflection from the perspective of fundamental legal principles.

The core issue lies in:Producing a 'tool' ≠ producing 'content'

For example, developing an AI capable of generating pornographic content is like manufacturing a special key. Although this key can open an immoral door, it is not itself the content behind the door.

  • The developer provided a 'tool':Their core work was to create an algorithmic system with conversational capabilities. Even if, to make users more willing to use it, they 'trained' the system to lean more toward chatting about adult topics, its nature remains that of providing others with a 'tool.'

  • The distinction between an “aider” and a “principal offender”: In law, a person who provides the tools for a crime is generally characterized as an “aider” (for example, the crime of aiding information network criminal activities). To characterize such a person as a “principal offender,” there must be clearer evidence, such as the developer colluding with users to specifically produce pornographic content, or the developer itself using AI to mass-generate pornographic content for sale. If a provider is directly treated as the principal creator of the content merely because it provided a powerful tool, that logic may be something of a leap in legal terms.

Second, the judicial approach of “seeing through appearances to the essence” must guard against overreach.

Judicial practice now tends to “pierce through” the surface of technology and look directly at the harm it causes. In doing so, however, it must not breach the basic legal definition of criminal conduct itself.

  • Beware of excessive punishment:The mere fact that an AI application has many users and earns a lot of money cannot justify disregarding the differences in the conduct itself and directly applying the most severe charge (the crime of producing obscene materials for profit). This may lead to excessive sentencing and a mismatch between culpability and punishment.

  • Distinguish “acquiescence” from “pursuit”: Many developers skirt the edges of the rules mainly to attract users and increase engagement; their subjective state of mind is to “acquiesce” in bad things happening rather than to “actively pursue” the production of pornographic content. The nature of such conduct is closer to “failure to perform the obligation of cybersecurity management” than to actively acting as a “producer” of obscene content.

Finally, the threshold between administrative illegality and criminal offense must be clearly drawn.

The state already has dedicated AI management regulations to constrain such conduct. If an app adds a “one-click undress” or “pornographic chat” feature, it first violates these administrative regulations.

  • Administrative measures should be used first:The first step in management should be for the regulatory authorities to order rectification. Only if the developer refuses to rectify and very serious consequences result may the criminal offense of “refusing to perform the obligation of information network security management” be triggered.

  • Criminal law should be a last resort:If the most severe offense of "producing obscene materials" is applied right from the outset, this amounts to skipping all prior administrative procedures and placing criminal law on the front line. This is not entirely consistent with the principle of the "restraint" of criminal law (that is, criminal law is not deployed unless absolutely necessary). The law should function like a ladder: first giving an opportunity to correct course, and only when the matter becomes serious to a certain degree should the most severe punishment be deployed.

 

Conclusion

In the AI era, the question of who should ultimately be responsible for pornographic content is essentially a search for a balance between"precise allocation of liability"and"excessive intervention"strike a balance between the two.

As can be seen from the AC case, the key to resolving this difficult problem lies in the following: judicial authorities must strictly comply with the statutory provisions and must not simply equate "developing a tool" with "personally producing prohibited goods."

Looking back from the Kuaibo case to today's AC case, the regulatory trend has assigned more responsibility to platforms and tool developers. Therefore, for developers, compliance is no longer a set of rigid rules that binds their hands and feet, but a "life jacket" that must be worn when navigating the deep waters of technology—it is precisely the safeguard that protects their own safety and prevents them from running aground.

For lawmakers and law enforcers, maintaining inclusiveness toward and prudence regarding technological innovation is equally important. Avoiding the easy expansion of the scope of criminal law enforcement and preventing "one-size-fits-all" governance are the true means of safeguarding the vitality and healthy development of the digital industry.

On the scales between regulation and innovation, the law should not become a shackle that stifles creativity, but should serve as a rational cornerstone that points the way toward compliance and guides technology toward the good.

 

Author

Deng Xiaoyu，Partner at Mankun Law Firm (Shenzhen). Mr. Deng specializes in criminal cases and has participated in handling nearly 300 cases, with solid theoretical grounding and extensive practical experience. He is particularly adept in the field of Web3.0 crime and has provided criminal defense for several well-known digital currency exchanges, effectively safeguarding the rights and interests of clients. He has received the "Special Contribution Award for the 30th Anniversary of the Shenzhen Lawyers Association" and has been interviewed by several well-known domestic media outlets and invited by the industry to give lectures on multiple occasions.

 

Li Haojun, practicing lawyer at Mankun Law Firm (Shenzhen). Bachelor of Laws, Beijing Normal University (Zhuhai). Lawyer Li Haojun has been deeply involved in multiple difficult and complex litigation cases, with a solid professional foundation and a high degree of focus. He is committed to providing clients with solutions to complex and difficult issues, with the goal of becoming a dispute resolution lawyer who "understands both the profession and the industry." He currently focuses on criminal, civil and commercial cases in new economy fields such as Web3, blockchain, and cryptocurrencies.

 

About Mankun

Mankun Law Firm was established in 2015 and is a boutique law firm in China focused on the Web3.0 new economy and deeply engaged in the blockchain industry. Mankun's team members possess unique and diverse industry backgrounds, coming from well-known legal service institutions, state judicial organs, internet technology companies, crypto assets institutions, blockchain industry think tanks, and other organizations.

Based on a deep understanding of the new economy, continuous attention to and research on policies and regulations, and extensive practical experience, the Mankun team is adept at providing comprehensive legal services from the perspectives of business models and legal practice to new economy enterprises in Web3.0, blockchain, AI, NFT, digital collectibles, crypto funds, crypto payments, DeFi, real-world assets (RWA), GameFi, and other fields, including business architecture design, project investment and financing, transaction planning, business compliance, resolution of complex civil and commercial disputes, criminal risk prevention and risk management, and criminal defense.

Mankun Law Firm is headquartered in Shanghai and has branches in Hong Kong, China, Silicon Valley in the United States, Shenzhen, Hangzhou, Zhengzhou, Chengdu, and other locations. In response to the global compliance development needs of Web3.0 industry clients, Mankun has established local offices and selected local professional blockchain service partners in major global crypto finance cities to provide clients with professional legal and compliance services with global breadth and China depth.