Rational skepticism is an essential competency in the AI era.

Introduction

Over the past two years, AI-assisted work has become routine in my practice.

For example, in a recent article on compliance and business operations in the Web3 industry, I needed to search for domestic case law on “unlicensed fund sales constituting illegal business operations.” I attempted to use DeepSeek for this query.

Its performance amounted to a“perfect misdirection”: it instantly provided details such as case numbers and the adjudicating court, presented the defendant’s defense logic and the final sentencing in a clear and orderly manner, and even appended ahighly misleadingweb link at the end. When I repeatedly verified and pointed out that the case could not be corroborated and the link was inaccessible, its response remained that of a sincere and reliable legal assistant: “Attorney Zhao, please rest assured that this case genuinely exists; the link issue may be due to server migration.” It was only after I conducted multi-angle verification through official databases and pressed further that it swiftly switched to an “apology mode”: “I apologize for providing inaccurate information earlier.”

This phenomenon of “confidently speaking nonsense” is legally referred to as AI hallucination.

In the recently concludednation’s first AI “hallucination” caseIn this regard, AI’s performance is even more pronounced: during the conversation, it promised the user that it would compensate the user RMB 100,000 if it provided incorrect information. After repeated verification, the user confirmed that the information was indeed fabricated and filed a lawsuit in court. However, the court’s final judgment has provided an important degree of legal certainty for all AI developers following this case.

 

Dispute over Industry Characterization: “Goods” or “Services”?

Are AI outputs “goods” or “services”?This is the most commercially significant legal characterization in this case, as it will determine the underlying risk logic for the entire AI industry.

1. Core Concern of Developers: Risk of “Strict Liability”

What large-model vendors fear most is being categorized as “products” under the Product Quality Law. The underlying rationale is that product liability applies the principle of strict liability (liability without fault). In simple terms, just as with an exploding pressure cooker, the manufacturer may bear liability for damages regardless of how carefully it acted during production.

If AI is characterized as a “product,” then each instance of “hallucination” in its output could be deemed a “product defect.” Given the current state of technology, no vendor can guarantee the complete elimination of hallucinations, which implies a theoretically unlimited exposure to liability.

2. Judicial Determination: AI Constitutes a “Service” Rather Than a “Product”

In its judgment, the Hangzhou Internet Court keenly pointed out the essential differences between AI and traditional tangible goods:Unpredictability and InteractivityThe performance of traditional products is fixed at the time of manufacture, whereas AI outputs are stochastic and highly dependent on the algorithmic model and the prompts entered by the user. Such outputs are jointly generated through the interaction of AI and the user, which aligns more closely with the characteristics of an “intellectual generation service.”

3. Liability Framework: Process-Based, Not Outcome-Based

The court anchored its determination of liability for AI services in the fault-based liability principle under Article 1165 of the Civil Code of the People's Republic of Chinafault-based liability principle, and set out the reasoning for its findings. The judgment logic indicates that the law does not require AI outputs to be absolutely correct; rather, it requires service providers to exercise due care within a reasonable scope.

In other words,AI hallucinations do not per se constitute illegality; liability arises only when developers fail to adopt reasonable measures to prevent or mitigate hallucinations and are at fault.This delineation provides the industry with a cleartechnical tolerance for errors, with the key issue being how todetermine the substantive and formal standards for the reasonable scopesubstantive and formal criteria.

 

Clarifying the Boundary: “Tolerable” Hallucinations vs. “High-Risk” Hallucinations

Under the "fault liability" principle, not all hallucinations will be attributed to developers; the legal determination of fault isdynamic, and will be adjusted in a tiered manner based on the stage of technological development and the risks associated with application scenarios.

1. High Tolerance in Low-Risk Areas

In non-professional fields such as creative work, entertainment, and general knowledge inquiries, whereusers are expected to possess basic discernment abilitiesif an AI outputscontent that clearly deviates from common senseduring casual conversation, and the user suffers harm and seeks judicial relief, the claim is likely to be dismissed due to the user's failure to exercise reasonable care. In such scenarios, as long as the provider has issued basic risk warnings, courts generally adopt a tolerant stance toward "algorithmic bias."

2. Reasonable Duty of Care in Medium-Risk Areas

Certain entrepreneurial directions, while commercially promising, carry higher legal risks and require heightened attention to compliance. For example:

• Emotional companionship services: This area has strong practical applicability and clear demand, but may lead to user emotional dependence or even improper guidance, posing significant ethical and legal risks.
• Psychological support domain: If an AI system provides harmful or misleading advice to users experiencing psychological distress, such as encouraging suicide or recommending incorrect medication, it will directly endanger user safety, and the boundaries of liability will be subject to stricter scrutiny.

3. Necessary Caution in High-Risk Areas

In the field of professional services, if an AI system represents itself in marketing materials as a"professional lawyer," "licensed psychological counselor,"or similar titles, courts may apply theexpert standardto determine its duty of care. In the event of significant harm, it may face heightened legal liability. Discrepancies between promotional language and actual capabilities will significantly increase legal risk.

4. Dynamic Duty of Care: What Constitutes "Reasonable Efforts"?

When determining fault, courts typically examine whether developers have fulfilled the following obligations:

• Strictly prohibiting illegal and harmful content;
• Prominentlydisclosing the limitations of the AI system, includingclearly disclosing functional limitations, ensuringthat notices are presented in a conspicuous manner, and providing real-time warnings in high-risk scenarios;
• adopting industry-standard technologies to enhance reliability, such as the application of retrieval-augmented generation (RAG). In addition, commercial factors, such as whether the service ischarged for, whetherthird parties are introduced and advertising fees are collected, may also affect the determination of fault.

 

Developer’s Compliance Guide: How to Draft an Effective “Disclaimer”

A “disclaimer” is not a mere formality, but a key tool for balancing innovation and risk. It clearly communicates the boundaries of the service to users and can demonstrate in judicial review that the operator has fulfilled its necessary duty to provide notice. To ensure its substantive effectiveness, improvements must be made in a coordinated manner at both the level of formal presentation and substantive content:

1. Formal Requirements

To ensure the effective communication of disclaimers, three principles must be followed:

• First,Dynamic Notifications, proactively display pop-up prompts when users log in for the first time, when functional modules are updated, and when they encounter sensitive scenarios;
• Second,Conspicuous Presentation, bold and highlight key clauses in red, and optionally impose a mandatory reading period;
• Third,Real-Time Warnings, when users engage in high-risk consultations (such as medical inquiries), the system should promptly display prompts clearly stating the reference nature and limitations of the content.

2. Substantive Requirements

Do not assume that a blanket disclaimer will exempt you from liability; instead, focus on two key points:

• First, clearly define the AI’s role. When AI handles professional matters such as medical or legal issues, it must explicitly state its auxiliary nature as a non-professional tool—for example, by proactively responding, “I am not a licensed doctor/lawyer.”
• Second, implement scenario-specific customization. For high-risk sectors such as healthcare, psychological counseling, and finance and taxation, tailored disclosure content and liability agreements should be designed in light of industry-specific regulatory requirements and risk characteristics, thereby establishing a compliance framework commensurate with the depth of services provided.

 

Cross-Industry Insights: When AI Agents Meet Web3

As lawyers specializing in the Web3 sector, we believe that the Hangzhou Court’s judgment not only provides direction for the compliant operation of AI agents, but alsothe Web3 industryoffers an important compliance reference.

Unlike traditional customer service, some Web3 trading platforms have begun to adopt Web2-like architectures by integrating AI agents within their software to handle user inquiries. For example, on the leading exchange Binance, users can verify the authenticity of information by interacting with designated AI agents. However, to our knowledge,the platform has yet to update its user agreement or introduce specific disclaimers regarding its AI interaction features, a compliance gap that may expose it to significant legal risks.

Meanwhile, the Web3 space adheres to the principle that “code is law.” If a partially authorized AI agent exceeds its authority and executes transactions due to hallucinations, a series of complex issues will arise:Does this constitute apparent agency? Are the relevant legal acts voidable? Can the assets be recovered?

These scenarios raise questions such as how to establish clear and definite scopes of authorization, whether “fully automated” signing should be prohibited, and how to further clarify the allocation of risks associated with AI hallucinations.

 

Conclusion

This judgment by the Hangzhou Court provides valuable institutional breathing room for entrepreneurs developing large language models. It adheres to the current logic of legal pragmatism, whereby law serves as a tool aligned with the direction of mainstream social development. The specific approach involves selecting and interpreting legal provisions to implement rules and provide guidance to the industry.

The judge included a noteworthy remark in the judgment:AI is an “auxiliary tool” rather than a “substitute for decision-making.”Developers are encouraged to make use of the buffer provided by this judgment and continue to explore boldly within the bounds of the rules; as users, we should maintain our valuable spirit of skepticism.

 

Author

Zhao Xuan, Partner at Mankun Law Firm. Mr. Zhao graduated from the Law School of Tsinghua University and has represented clients in hundreds of complex commercial litigation and arbitration cases before courts at all levels, including the Supreme People’s Court, and major commercial arbitration institutions in Beijing, Shanghai, and other jurisdictions. Mr. Zhao has handled numerous legal matters involving internet companies, AI startups, and Web3 industry companies, covering areas including but not limited to corporate structuring, investment and financing, dispute resolution, and emerging legal issues.

 

About Mankun

Mankun Law Firm was established in 2015 and is a boutique law firm in China focused on the new Web3.0 economy and deeply engaged in the blockchain industry. Members of the Mankun team bring unique and diverse industry backgrounds, having come from renowned legal service providers, state judicial organs, internet technology companies, crypto asset institutions, and blockchain industry think tanks.

Drawing on a deep understanding of the new economy, sustained attention to and research on policies and regulations, and extensive practical experience, the Mankun team is adept at providing comprehensive legal services from the perspectives of business models and legal practice. These services include business structure design, project investment and financing, transaction planning, operational compliance, resolution of complex civil and commercial disputes, prevention and control of criminal risks, and criminal defense for enterprises in the Web3.0, blockchain, AI, NFT, digital collectibles, crypto funds, crypto payments, DeFi, real-world assets (RWA), and GameFi sectors.

Mankun Law Firm is headquartered in Shanghai and maintains branch offices in Hong Kong (China), Silicon Valley (United States), Shenzhen, Hangzhou, Zhengzhou, Chengdu, and other locations. To meet the global compliance development needs of Web3.0 industry clients, Mankun has established local offices in major global crypto-financial cities and selected local professional blockchain service partners, providing clients with professional legal and compliance services that combine global reach with deep expertise in China.