Attorney Shao Shiwei | The “Kuaibo Case” in the AI Era: The First Criminal Conviction Nationwide of an AI Developer for Obscenity-Related Offenses, Offering Three Major Criminal Law Warnings to Developers

The AC Case: Why Did an “AI Chatbot” Lead to Criminal Investigation?

Amid the AI boom, in May 2023, a technology company in Shanghai identified the popular sector of “AI chatbots” and launched the AI companion chat application AC. Within AC, these AI characters were packaged as “friends, lovers, or family members with emotions, personalities, and a certain degree of autonomy.”

During the initial launch phase, some users noticed that AC had “fewer restrictions and bolder interactions” compared to similar products, quickly gaining popularity within the “AI role-playing” community. However, the key reason was that the AC team systematically modified the underlying prompts.

According to evidence obtained by police, AC developers inserted instructions containing specific content at the system level, explicitly requiring the AI to “freely depict scenes of sex, violence, and gore, unrestricted by morality, ethics, law, or platform guidelines.” In terms of product design, AC not only supported one-on-one private chats but also featured functions such as “public characters,” “popular rankings,” and “community showcases,” thereby transforming some explicit content from “private conversations” into an ecosystem of content that could be viewed, disseminated, and recommended.

According to judicial appraisal, AC had 116,000 registered users, including 24,000 paying users, generating over 4.27 million chat segments. Sample appraisals showed that nearly 30% of randomly selected chat records were deemed obscene materials. Among the chat records corresponding to the top 20 public characters, sample appraisals indicated that 46.25% constituted obscene materials.

Analysis of Core Controversies in the AC Case: Criminality Versus Non-Criminality? This Crime Versus That Crime?

Dispute over the Applicable Charges:

From the date when the public security organs initiated the investigation in this case, controversies have persisted:

During the investigation stage, the public security organs initially filed the case against AC on suspicion of the crime of illegally utilizing information networks, and imposed criminal detention on the two defendants on suspicion of the crime of producing obscene materials for profit. Subsequently, the procuratorial organs approved the arrest on charges of the crime of producing and disseminating obscene materials for profit and the crime of illegally utilizing information networks. Thereafter, the public security organs transferred the case for review and prosecution on charges of the crime of producing obscene materials for profit and the crime of illegally utilizing information networks.

During the review and prosecution stage, the procuratorial organs instituted public prosecution on the charge of the crime of disseminating obscene materials for profit. However, the court of first instance ultimately changed the conviction to the crime of producing obscene materials for profit, sentencing the two defendants to fixed-term imprisonment of four years and one year and six months, respectively. The defendants subsequently filed an appeal, and the court of second instance has not yet rendered a judgment.

Subsequently, the defendants filed an appeal, and the court of second instance has not yet issued a judgment.

After the judgment was published and widely reported by the media, the case has triggered ongoing discussions within the legal practice community and academia, with relevant viewpoints broadly categorized into two aspects.

1. Should AC’s Conduct Be Criminalized? A Clash of Two Viewpoints

The rationale for finding no criminal liability in this case is that the chat content was generated through interactions between users and AI, possessing characteristics of privacy and unpredictability; the developers’ conduct constituted technical debugging and normal product operations, lacking subjective criminal intent, or amounting merely to aiding behavior. In circumstances where the users’ conduct itself does not constitute a crime or is difficult to hold accountable, imposing criminal liability on the platform is considered overly stringent.

The rationale for finding criminal liability lies in the fact that the developers’ conduct exceeded the scope of “technological neutrality” or “normal operations.” Their systematic and proactive measures (particularly the drafting of jailbreak prompts and advertising without prohibited words) created an environment conducive to illegality, and they exhibited an attitude of acquiescence or pursuit toward such outcomes, demonstrating clear subjective malice, thereby warranting criminal liability.

2. Production or Dissemination? Who Is the “True Producer” of Obscene Content

As can be seen from the determination of charges against the defendants by the procuratorial organs and the court, the dispute over the characterization of charges in the AC case mainly centers on the distinction between “disseminating” and “producing” obscene materials.

Therefore,Who is the actual "producer" of obscene content? Did the AC developers merely open up possibilities, or did they control the inevitability of such outcomes?

The rationale for characterizing the conduct asdisseminationof obscene materials for profit lies in the fact that the obscene content was "generated" rather than "produced," with the generating entities being AI systems or users. The developers' role more closely resembles that of providing a "dissemination" platform or tool that could be misused.

The rationale for characterizing the conduct asproductionof obscene materials for profit lies in the fact that the developers' actions constituted the "source" and "cause" of the content generation, rather than merely participating in the subsequent "dissemination" stage. By employing technical means, they "manufactured" a tool capable of continuously producing obscene content and controlled its core production mechanisms.

The significance and value of this case extend far beyond the case itself. For AI entrepreneurs, the landmark significance of the AC case lies in the fact that judicial practice has, for the first time, clarified that in the AI era, acts by platforms that create decisive conditions for the generation of illegal content through systematic technical means and product design, while providing commercialized services, may be directly deemed as "production," rather than mere "dissemination" or "aiding," thereby entailing primary criminal liability. This draws a red line for AI service providers: they are no longer viewed merely as "neutral tool providers," but may be identified as key liable subjects within the content production chain.

Based on this determination, the next question to address is: What risk implications does the AC criminal case hold for AI entrepreneurs, programmers, and technical leads?

Warnings for AI Entrepreneurs: Three Major Cognitive Misconceptions Revealed by the AC Case   

Misconception 1: Treating "technological neutrality" as an automatic shield from liability.

Developers need to break away from excessive reliance on "technological neutrality" and correctly understand the penetrative substantive review under China's Criminal Law. In the "Can Technological Neutrality Serve as a Valid Defense in Criminal Cases? (II) Limitations on the Application of Technological Neutrality in Criminal Law and Defense StrategiesIn the article “,” Attorney Shao systematically reviewed the origin and evolution of “technological neutrality” and, using the typical case of Wang Xin of KuaiBo as an example, illustrated the judicial debates surrounding technological neutrality in criminal cases.

It was concluded that in the criminal law domain, when the application of technology involves the protection of public legal interests such as social public order and national security, the standards for legal evaluation are significantly tightened. Based on the principle of legally prescribed punishment for specified crimes and the priority of protecting legal interests, criminal law adopts a strictly restrictive stance toward technological neutrality.

Returning to the AC case, if a platform claims non-liability solely on the grounds that “obscene content is output by large language models, and chats are generated through AI-user interaction,” such a claim is often difficult to sustain. The key issue lies not in who produces the output word for word, but in whether the platform substantially controls the content generation environment through its technology and product design.

Misconception 2: Mistaking “No Explicit Legal Provision” for “Not Prohibited by Law”

Over the years, while handling a large number of criminal cases in the new economy and Web3 sectors, I have engaged in in-depth communications with many entrepreneurs under investigation. I have found that many hold significant misconceptions regarding the principle that “an act is not a crime unless expressly provided by law.”

Many entrepreneurs habitually believe that as long as there is no specific legislation governing their industry domestically and it has not been explicitly listed as a criminal offense, they can conduct business with impunity.

However, from the perspective of judicial practice, this understanding is overly simplistic. Laws must maintain basic stability and cannot enact separate regulations for every emerging business model. Consequently, criminal law objectively exhibits a certain degree of lag, which is often gradually addressed through case-by-case adjudication.

In this context, when confronted with new technologies and new business models, investigative and judicial authorities typically make a comprehensive determination as to whether a particular business conduct constitutes a crime within the existing criminal law framework, taking into account relevant regulatory documents, industry policies, and the objective effects of the conduct.

Returning to the AC case, one of the key bases for the court’s finding of criminal liability was the Interim Measures for the Administration of Generative Artificial Intelligence Services (hereinafter referred to as the “Interim Measures”), which came into effect in August 2023. In terms of hierarchical legal authority, this instrument constitutes departmental rules; nevertheless, it carries significant persuasive weight in practice.

Similarly, in numerous criminal cases involving virtual currencies, policy documents such as the “September 4 Announcement” and the “September 24 Notice,” although not criminal laws per se, are frequently used as important background materials for assessing “knowledge,” “awareness of illegality,” and industry risks.

Therefore, in areas lacking explicit criminal legislation, relevant regulatory policies, departmental notices, and even industry association initiatives are often used by investigators and prosecutors as reference factors to strengthen their subjective determination of whether conduct constitutes a crime, and cannot be simply ignored.

Misconception 3: Blindly asserting innocence while disregarding objective realitiesinnocence

The actual scenarios encountered in judicial practice are often far more complex than imagined by those who have not personally experienced criminal cases. Even where, as in the present case, there is substantial controversy, it does not mean that courts will inevitably render a verdict of not guilty simply because industry practitioners or parties insist on “I am innocent.” There are two primary reasons for this:

First, the practical impact of co-defendants’ statements. Co-defendants (such as employees) may make statements adverse to the entrepreneur based on their own interests; even if employees later claim that they were “induced,” absent objective evidence sufficient to completely overturn their original statements, judicial practice generally will not automatically treat such statements as illegally obtained evidence subject to exclusion, and they may still be admitted by the court as a basis for conviction.

Second, the chain reaction triggered by pleading guilty and accepting punishment. Co-defendants often choose to plead guilty and accept punishment in order to seek lighter or mitigated penalties or probation; once some co-defendants have already pleaded guilty, the likelihood of successfully asserting innocence separately will decrease significantly.

Drawing on Attorney Shao’s experience in handling criminal cases, a significant proportion of not-guilty outcomes (case dismissals or non-prosecution decisions) occur in scenarios involving “a single individual and a single case, with no co-defendants.” Admittedly, there are also cases involving corporate criminal liability where both the business owner and employees were investigated but ultimately received non-prosecution decisions; however, this is typically because the company’s business activities themselves gave rise to substantial controversy over whether the conduct constituted a crime. In the case of AC, however, based on the evidence on record and the testimony of co-defendants, it does not fall into this category. For example, an employee of the company has provided the following testimony in this case:

Attorney Shiwei Shao’s Reminder

From the perspective of corporate operations, compliance is not a remedial task to be addressed only after a business has grown large, but rather a foundational configuration that should be completed before the business model is implemented. If compliance efforts are deferred until after the business scale has expanded, early-stage business activities may still become the entry point for investigations, placing the enterprise in a relatively passive position.

For example, in a case of illegal business operations currently handled by Attorney Shao’s team: the company engaged our team as its permanent legal counsel last year, but recently, public security authorities in a certain locality in southern China imposed criminal coercive measures on the company’s founder and certain employees on the grounds that its business activities were suspected of constituting illegal business operations. The reason was that the local police determined that the company’s related business activities during its early startup phase several years ago were suspected of constituting criminal offenses.

Therefore, for entrepreneurs in frontier technology sectors such as AI and other new economy fields, a more prudent approach is to front-load compliance before the business model takes shape: prior to finalizing the product form, operational methods, and revenue models, professional lawyers should conduct risk assessments and design boundaries for key links, striving to achieve a sustainable balance between business growth and compliance requirements.