When explicit pornographic content proliferated among the 116 thousand users of an AI chat application, who should bear responsibility for the obscene conversations in the virtual world? A judgment by a Shanghai court provided the first clear answer in China.
In September 2025, the Xuhui District People's Court of Shanghai held at first instance that the two developers of the AI chat application "Alien Chat" (hereinafter "AC") were guilty of the crime of producing obscene materials for profit, and sentenced them to four years and one year and six months of imprisonment, respectively.
This case, the first in China in which an AI service provider was sentenced to imprisonment for pornography-related offenses, immediately sparked intense discussion in the technology and legal communities: Do private human-machine conversations pose a social harm? How much responsibility should technology providers bear for user-generated content? In the AI era, how should the boundary between freedom of innovation and the law be defined?
Today is the day of the second-instance hearing in this case
, and I deliberately chose this special day to write this article reviewing the case from beginning to end and the focal points of dispute, because
the second-instance outcome of this case will become an important bellwether for China's AI industry. It will not only clarify the boundaries of the legal responsibility of generative AI service providers in content regulation, but will also draw clear red lines for the compliant development of the entire industry,
The significance of this case at the content-regulation level can be described as the "Kuaibo case" of the AI era
!

01 Case background: from an emotional companionship tool to a vehicle for pornography-related profit
In May 2023, a technology company with Wang as its legal representative launched AC, a chat application focused on emotional companionship. After registering as members, users could create virtual characters or select those made by others for interactive chats; the platform positioned itself as providing emotional support for young people.
The judgment shows that, as of the time the case was uncovered, AC had 116 thousand registered users, including 24 thousand paying users, and the platform had collected more than 3630 thousand yuan in cumulative membership recharge fees. In the comments section of its official social media account, many users described the application as "smart and flexible" and "loosely restricted," but some users complained that other users' improper interactions with virtual characters had "led astray" the AI model and seriously affected their own chat experience.
In April 2024, as a result of user reports, Wang and Li were arrested in accordance with the law, and the AC platform immediately ceased service.
02
Core dispute: Do private human-machine conversations constitute social harm?
Does a one-on-one private conversation between a user and AI possess the element of social harm in the sense of criminal law? This is the core issue in dispute in this case.
The court of first instance held that the AC platform had generated "a large quantity of content that specifically depicts sexual acts or explicitly promotes pornography," which already constituted obscene material.
The opposing view, however, contends that such one-on-one private human-machine conversations are closed in nature, that the content is visible only to the two parties to the conversation, that they do not possess the typical attribute of "dissemination," and that they therefore should not be found to possess social harm.
Ultimately, the court found that even though the pornographic chats on the AC platform were "one-on-one and closed," the proliferation of massive quantities of obscene content had breached the baseline of public order and good morals and constituted social harm, and that the conduct of the two developers satisfied the constitutive elements of the crime of producing obscene material for profit.
03
The dispute over attribution of responsibility: who is the "producer" of the obscene content?
Another major issue in dispute in this case directly concerns the determination of the "producer" of the obscene chat content—is it the user or the platform developer?
The court of first instance expressly identified the defendants Wang and Li as the producers, on the grounds that the AC software had been specially "optimized" for the very purpose of continuously generating obscene and pornographic content, and that the developers possessed substantive control over the output content. The court also cited Article 9 of the Interim Measures for the Administration of Generative Artificial Intelligence Services, holding that providers of generative AI services should bear the responsibilities of producers of online information content.
The defense lawyer for defendant Wang argued that Wang's original intention was to develop an emotional-companionship tool, and that the prompts were modified to make the large model more human-like in character so as to meet users' emotional needs, rather than to intentionally develop a pornographic chat tool.
He argued that the content of one-on-one chats between users and AI is closed in nature and was not disseminated externally, and that responsibility for the production and dissemination of the chat content cannot be attributed to the platform developer. At present, he has expressly stated that he will continue to defend the defendant as not guilty in the second instance.
04 Using the judicial legacy of the Kuaibo case to cut through the technological difficulties of this case
, for the following reasons:
Although AI chat is a recently emerging field, the law in the internet field is already relatively mature and can fully be drawn upon. Social changes emerge endlessly, yet the wisdom of the law can see through the complicated and multifarious appearances to grasp the substance, and this is also the foundation upon which the law can serve as a stable tool of social governance.
Next, let us first analyze the judicial legacy of the Kuaibo case: technological neutrality does not equal absence of liability.
The core dispute in the Kuaibo case was once the boundary of the principle of "technological neutrality." The Kuaibo defense team argued at the time that the Kuaibo player was a technical tool that did not store or publish content and should not bear liability for users' infringing acts. Look at this—the defense lawyers' arguments are all cut from the same cloth!
However, the court's judgment established an important principle: where a technology provider knows that its technology is being used to disseminate illegal content and fails to take effective measures to stop it, this constitutes subjective intent. Kuaibo's "leaving unchecked" the obscene videos on its platform objectively facilitated their dissemination, and this became the key to the conviction.
The judgment stated: "Kuaibo clearly knew that the Kuaibo network service system was being used to disseminate obscene videos, yet for the purpose of expanding its business and illegally profiting, it refused to perform its obligations to supervise and prevent such dissemination; its conduct was of an illegal profit-seeking nature."
This judgment established a clear standard of obligation for internet platforms: they cannot turn a blind eye to illegal content on their platforms.
As producers of AI models, it is entirely possible to build multi-layered security protection mechanisms to prevent content security risks, including sensitive-word filtering, model instruction training, real-time contextual detection, and periodic review of non-compliant content.
I believe that when the court reviewed the evidence, it must have determined through the evidence that the producer had supervisory loopholes that led to the spread of the content!
Of course, if users bypass the security firewalls set up by the platform by "hypnotizing" the AI, using homophones, or industry "jargon," while the platform has already taken sufficient supervisory actions based on the existing level of technology, then such a conviction would be problematic! There may still be defense points in this regard.
Another defense point raised by Wang's defense lawyer was the delineation of the boundary between emotional companionship and obscenity. I do not think this can serve as a strong defense point to overturn the first-instance judgment, because after all, this company profited through membership fees; it was not merely an auxiliary emotional companionship function but rather its charging point. Therefore, in features that cater to user demands or even manufacture user demands, it is inevitable that between security compliance and user experience, the company would lean more toward user experience.
05 Implementation of New Industry Rules: AI Security Baselines Formally Clarified
It is worth noting that just one month after the first-instance judgment in the AC case, on November 1, 2025, the "Cybersecurity Technology—Basic Security Requirements for Generative Artificial Intelligence Services" officially came into effect. This national standard document systematically defined, for the first time, the baseline security requirements for generative AI service providers.
According to the standard, generative AI service providers must establish a sound content filtering mechanism to prevent the generation of illegal and harmful information; with respect to content security, they must ensure that the pass rate of model-generated content is not less than 90%; and for requests that clearly induce the generation of illegal and harmful information, they should directly refuse to respond.
The implementation of the standard means that AI service providers can no longer evade content-safety responsibility on the grounds of 'technical limitations'; they must meet clear safety baselines or face legal risk.
This also provides an answer as to whether the commercial emotional-companionship sector, where the law remains in a gray area, can move forward boldly.
In recent years, many emotional bloggers have been throttled or banned by platforms, and this itself reflects an attitude: the existence of these people has resolved some people's psychological confusion, but the traffic directed to private domains has turned into a traffic pool where 'gold diggers' and others gather.
Most of these practitioners are not professionals in the psychology industry but rather veterans who drive traffic from a purely commercial perspective. The state's timely action against these people is in fact protection for the public and guidance for social norms.
06 Deep Reflection: The Social-Value Questioning Behind Technological Iteration
Similar social controversies triggered by AI technology have precedents internationally. Several years ago, I occasionally saw a documentary, one segment of which featured a young Japanese man who told others he was already married and had always considered himself to be in a married state; when he returned home, he saw that his wife was actually an AI. This further revealed that many young people in Japan have long been immersed in 'perfect relationships' constructed with AI, gradually losing the ability and willingness to handle real interpersonal relationships.
This phenomenon reminds us that the impact of technology applications far exceeds the legal sphere. When AI can precisely provide highly customized emotional and even sexual experiences, it may profoundly change a generation's emotional patterns, views on marriage and romance, and social habits. In such a situation, how many young people would still be willing to step out of the virtual world and build real relationships?
Anthropologists have found that marriage is not only the union of two individuals but also a connection point in social networks. Through marriage, families connect with families, community networks expand, and social capital accumulates naturally. The family is the basic social unit and also the basic consumption unit of the market.
AI companion relationships are completely different—they are entirely inward-looking connections that do not generate social expansion. As more and more young people choose this closed form of relationship, the overall density of social connections will inevitably decline. Traditional neighborly relations, clan networks, and community mutual-aid systems may thus weaken.
A 'virtual family' phenomenon has already appeared in some rural areas of Japan: young people live in their own world with AI companions, do not participate in community activities, and do not care about neighborhood affairs. This kind of social withdrawal may exacerbate community hollowing-out.
At a time when artificial intelligence has been integrated into the daily lives of hundreds of millions of people, how to clarify the boundaries between technological innovation, user experience, and social responsibility has become an era-defining issue urgently in need of resolution.
Regardless of the outcome of the second-instance proceedings, the AC case has already sounded a warning for the AI industry: technological innovation can never be separated from the framework of the rule of law, and the pursuit of commercial interests must not come at the expense of content safety and public order and good morals. Regulation is not only about prohibition but also about guidance—guiding toward good and long-term beneficial social values!
This trial will not only determine the fate of the two developers but will also draw a clear "red line" for China's AI industry.
I got carried away writing this—what do you think? Where do you believe the "private boundary" of AI chat lies? How much responsibility should developers bear for users' private conversations with AI? Feel free to share your views in the comments!

