In April this year, Tencent Cloud EdgeOne published an observational article stating that in its AI crawler handling data, AI crawler traffic to e-commerce websites ranked first among all industries. One coupon-aggregation shopping-guide platform was hit by AI crawlers nearly 200,000 times in a single day. This is the vendor's account, not official statistics, but the direction is broadly credible: the approach to blocking crawlers is shifting from maintaining a blacklist to determining, based on behavior, who this visitor actually is and what it wants to do.
For teams building AI products, this cuts both ways. Your agents, price-comparison features and product-selection models most likely source their data through crawlers; at the same time, your own site, documentation and pricing pages are being read by other people's crawlers. An old question is therefore being asked again: what happens if I don't comply with that robots.txt file in the website root directory?
I have gone through the judgments of recent years. The answer is not a single sentence but a ladder, running from being unable to recover any money at all all the way to a criminal conviction.
Violating robots.txt does not support a claim for breach of contract
First, let me make clear what this file is.
In 1994, the Dutch engineer Martijn Koster proposed that a website could place a text file in its root directory telling crawlers which directories need not be crawled. In June of the same year, a group of people who wrote crawlers formed a "Robots Exclusion Standard" in a mailing list. The document was candid: it was not an official standard registered by a standards body, had no mandatory force, and could not guarantee that all current or future crawlers would use it. In 1997, Koster submitted it to the Internet Engineering Task Force in an attempt to make it an industry standard; it has not been adopted to date.
The above is not my summary; it is the facts ascertained by the Beijing High People's Court in its judgment.
Its syntax is even more telling. The default value is "allow": a website that has not placed a robots.txt, or whose file content is empty, is open to all crawlers; only to prohibit is a dedicated Disallow line required. The original design intent of this file is guidance, not a wall.
Therefore, under Chinese law, it does not constitute a contract.
A contract requires an offer, an acceptance and the parties' mutual assent, and when your crawler visits a web page, no link in that chain ever reaches mutual assent. This is like a restaurant posting "No outside drinks" at its door: if you bring your own, the establishment can ask you to leave, but it cannot sue you for breach of contract, because the two sides never signed a contract at all.
One boundary to add: platform agreements and robots.txt are two different things. The "I have read and agree" you clicked when registering an account—that user agreement has contractual effect, and violating it is a separate issue.
What the court actually examines is how you use this notice
If not through contract, then through what?
On July 3, 2020, in the second-instance judgment in Qihoo v. Baidu, the Beijing High People's Court settled the matter definitively:
The core of the dispute in this case is not the validity of the robots protocol itself, but whether the use by Baidu Netcom and Baidu Online of the robots protocol at issue constitutes unfair competition under Article 2 of the Anti-Unfair Competition Law.
The court does not assess whether the protocol is valid; it examines only what the parties did with it.
The anchor is Article 2 of the Anti-Unfair Competition Law:
Article 2: In production and business activities, business operators shall follow the principles of voluntariness, equality, fairness and good faith, comply with the law and business ethics, and participate fairly in market competition.
Unfair competition as referred to in this Law means conduct by a business operator in production and business activities that violates the provisions of this Law, disrupts the order of market competition, and harms the lawful rights and interests of other business operators or consumers.
The background of the case is the 3B battle of that year. Baidu's robots protocol adopted a whitelist system, and 360 Search was never included after its launch; Qihoo sued, and both instances held that Baidu constituted unfair competition and awarded RMB 200,000 in damages. In the case in the opposite direction, Baidu sued 360 for forcibly crawling in disregard of robots, and the first instance ordered Qihoo to pay RMB 700,000. The same protocol, and unfair competition was found in both directions.
The judgment also contains an analogy, written by the judge in plain language:
If a website is compared to a museum open to the public, the robots protocol is equivalent to a notice board hung at the museum entrance, informing visitors which areas are not open to the public... The content of the notice board should be the same for all visitors; if a certain category of persons is to be prohibited from entering, there must be reasonable and proper grounds.
Following this analogy, the court listed three acceptable proper grounds: protecting the website's internal or sensitive information, maintaining the normal operation of the website, and protecting the public interest. Beyond these, shutting people out by reference to the business entity rather than the information content is targeted and discriminatory.
Blocking you is not unlawful; it is only unlawful when you crawl the content away and use it as a substitute
When this yardstick is applied outside search engine scenarios, the scale changes.
In April 2016, Weibo singled out Toutiao's crawler, ToutiaoSpider, for blocking in the robots.txt of m.weibo.cn, while other crawlers continued to crawl without hindrance. ByteDance sued for RMB 100 million in damages; the Beijing Intellectual Property Court at first instance held that Weibo had engaged in unfair competition and awarded RMB 300,000. On October 8, 2021, the Beijing High People's Court, on appeal, reversed the judgment in its entirety and dismissed all of ByteDance's claims.
There were two layers to the reasons for the reversal. First, the appellate court held that the Self-Discipline Convention on Internet Search Engine Services, which the first instance had treated as the basis for commercial ethics, constituted commercial ethics only for the search engine service industry and could not be treated as commercial ethics generally applicable across the entire internet industry. Second, and more importantly, the following passage:
Search engines bring traffic and benefits to the websites being searched, whereas web robots that are not search engines often do not bring traffic to the websites being searched and may instead take traffic away from them.
After a search engine crawls, it sends users back; after an AI crawler crawls, it directly provides answers, and users need not visit the original site again. Under this distinction, AI crawlers stand on the side of "taking traffic away." Therefore, where a website unilaterally blocks a particular AI crawler, so long as it does not harm consumer interests, the public interest or the competitive order, this in principle falls within the enterprise's autonomous business discretion.
The same pair of parties was involved in another case. On May 17, 2021, the Beijing Haidian District People's Court held that ByteDance's direct transfer of crawled Weibo content into "Weitoutiao" constituted unfair competition.
Read together, the two judgments lead to a fairly clear conclusion: if I block you, that is not unlawful; if you crawl my content and make a product that replaces mine, that is unlawful.
The Beijing High People's Court ultimately left a caveat: this does not mean that any robots protocol set by an internet enterprise is necessarily justified by virtue of autonomous business discretion.
Complying with the protocol may still cost tens of millions in damages
Reading this far, one may easily draw an erroneous conclusion: then if I honestly crawl in accordance with what robots.txt says, surely there will be no problem.
The case that concluded on second instance at the end of last year provides the answer. The "Zhanying" public opinion product operated by Wangzhi Tianyuan crawled Tencent News data, accumulating more than 110,000 items, provided full-text browsing and analysis reports within its own App, and charged SVIP users and enterprise clients annual fees ranging from RMB 4,998 to RMB 160,000. Tencent claimed RMB 30 million in damages.
The defendant's core defense was precisely this: the crawler protocol published by Tencent expressly permits other operators to crawl news data, and we acted in accordance with the protocol. The response of the Fuzhou Intermediate People's Court at first instance was the sharpest statement in the data crawling field in the past two years:
The court made clear that data circulation should follow the principle of "authorization plus restriction"; the fact that Tencent permitted data scraping in order to safeguard public access to information does not mean it permitted for-profit commercial use.
The first-instance judgment awarded RMB 2 million in economic losses and more than RMB 280,000 in reasonable expenses; the Fujian High People's Court, on second instance, dismissed the appeals of both parties and upheld the original judgment.
Permission to scrape answers only "whether the data may be obtained," not "what may be done with it once obtained."
The latter question is answered by platform agreements, competitive order and commercial ethics.
Another figure is even more startling. Amap's "congestion delay index" was publicly available on its website; Wind obtained it by changing IP addresses and forging browser identifiers and provided it to paying users through the WIND financial terminal, arguing that it was public information and manually collected, but the court did not accept this. The Beijing Chaoyang District People's Court, at first instance, awarded RMB 12.5 million in damages, and the Beijing Intellectual Property Court upheld the judgment on appeal, with the damages exceeding the statutory cap.
Being publicly displayed does not mean it may be taken for free and resold.
The new law moves the red line from declarations to technology
The above judgments all applied the old law, relying on the general clause in Article 2 as a fallback. Since October 15 last year, the situation has changed.
The second amendment to the Anti-Unfair Competition Law took effect, moving the internet-specific provision from Article 12 of the old law to Article 13 and adding a new paragraph specifically governing data:
Business operators shall not, by improper means such as fraud, coercion, or circumventing or destroying technical management measures, obtain or use data lawfully held by other business operators, harming the lawful rights and interests of other business operators or disrupting the order of market competition.
Penalties are provided as well:
Article 29: Where a business operator, in violation of the provisions of paragraphs 2, 3 and 4 of Article 13 of this Law, uses the internet to engage in unfair competition, the supervisory and inspection department shall order it to stop the illegal act and impose a fine of not less than RMB 100,000 and not more than RMB 1 million; if the circumstances are serious, a fine of not less than RMB 1 million and not more than RMB 5 million shall be imposed.
This provision sets out the wrongfulness of the means as an explicit category, so there is no longer any need to start from commercial ethics in every case. What truly requires close attention is those six characters: technical management measures.
Login walls, permission verification, interface signature verification, risk management and human-machine verification, rate limiting, and paywalls all count, because they physically block you. robots.txt cannot block a single access; your crawler ignores it and still retrieves all the content. It is a statement, not a device. The Beijing High People's Court put it plainly in the Weibo case: regardless of whether it is complied with, the robots protocol does not produce the result of mandatorily prohibiting access.
This judgment shares the same origin as copyright law. Article 49, paragraph 3 of the Copyright Law sets an "effective" threshold for technical measures:
The term "technical measures" as used in this Law means effective technologies, devices or components used to prevent or restrict the browsing or appreciation of works, performances, sound recordings or video recordings, or the provision of works, performances, sound recordings or video recordings to the public through information networks, without the permission of the right holder.
A single line of notice text does not meet this threshold.
So, by comparison, the new law punishes picking the lock, not ignoring the notice.
In the Amap case, changing IP addresses and forging browser identifiers were standard lock-picking conduct.
Once that mechanism is crossed, it is no longer a matter of paying compensation
One level higher, the nature changes.
Article 285, paragraph 2: Whoever, in violation of State regulations, intrudes into a computer information system other than those specified in the preceding paragraph, or uses other technical means to obtain data stored, processed or transmitted in that computer information system, or exercises illegal control over that computer information system, and the circumstances are serious, shall be sentenced to fixed-term imprisonment of not more than three years or criminal detention, and shall also, or shall only, be fined; if the circumstances are especially serious, shall be sentenced to fixed-term imprisonment of not less than three years but not more than seven years, and shall also be fined.
"Serious circumstances" is not a vague term; Judicial Interpretation No. 19 of 2011 provides the figures:
Article 1: Whoever illegally obtains data from a computer information system or illegally controls a computer information system, and any of the following circumstances exists, shall be determined to constitute "serious circumstances" as provided in Article 285, paragraph 2 of the Criminal Law:
(1) obtaining identity authentication information for online financial services such as payment settlement, securities trading, and futures trading, in ten or more sets;
(2) obtaining identity authentication information other than that specified in item (1), in five hundred or more sets;
(3) illegally controlling twenty or more computer information systems;
(4) obtaining illegal gains of 5,000 yuan or more, or causing economic losses of 10,000 yuan or more;
(5) other circumstances of a serious nature.
Illegal gains of 5,000 yuan; economic losses of 10,000 yuan.
This threshold is so low that many startup teams have no idea of it.
In the case known as the first in the country in which web scraping led to criminal punishment, a company in Shanghai used web scraping to capture video data from another party's servers; the court convicted the defendant entity of the crime of illegally obtaining computer information system data and imposed a fine of 200,000 yuan, and the four defendants were each sentenced to terms ranging from one year to nine months. The case in Changzhou, Jiangsu involving the illegal acquisition of data through AI technology is closer to the present day: the defendant entity's AI interactive platform bypassed the encryption algorithm of a certain community e-commerce app, forged requests to capture user data, and cracked the private message interface to place advertisements for paying merchants, obtaining illegal gains of more than 6 million yuan.
There is another point that is easily overlooked. Guiding Case No. 36 of the Supreme People's Procuratorate confirms that using an account and password to log in to a computer information system beyond the scope of authorization constitutes the act of intruding into a computer information system. This point is especially important for teams building Agents: where your Agent uses an account and password provided by a user to log in to a platform, the user's authorization does not mean the platform has authorized it.
Three types of companies, and what each should change
For parties building AI products who need to scrape data, the focus of compliance should shift from "whether robots has been complied with" to two areas.
As to means, none of the four mechanisms—login, signature verification, risk management, and rate limiting—may be bypassed; bypassing any one of them changes the nature of the conduct from a commercial dispute to picking a lock, and in serious cases to a criminal case. As to purpose, do not create a substantial substitute for the original site, and do not provide the scraped dataset itself to outside parties for consideration. Then add one evidentiary habit: before scraping, save a snapshot of the target site's robots.txt, and during the process retain frequency logs; if you are actually sued, this is your basis for proving that you neither circumvented anything nor overwhelmed the other party's servers.
The party being scraped—for example, e-commerce platforms and content sites—must accept a reality: robots.txt on its own is basically useless; its role in litigation is to prove that you expressed an intent, not to stop anyone. There are two legally meaningful actions. First, embed the restrictions in technology—login walls, API authentication, rate limits, risk controls—so that once deployed, a party that bypasses them directly engages Article 13, Paragraph 3 of the new law and even Article 285 of the Criminal Law. Second, state expressly in the platform agreement that unauthorized commercial use is prohibited; this clause was key to Tencent prevailing in the Zhanying case. In addition, blocking specific crawlers is in principle a matter of autonomous business operation, but where discriminatory distinctions are made by business entity without legitimate justification, there remains a risk of being sued.
The middle group—Agent service providers and data service providers—has a foothold on neither side: upstream, they scrape others' data on behalf of clients; downstream, they deliver data to clients. The authorization chain must be traceable all the way to the end user, and the contract must clearly allocate responsibility for the data source and the scope of use. Do not use technological neutrality as a shield; the Fuzhou Intermediate People's Court specifically stated in the Zhanying case that review of legality must break through the formal constraints of technological neutrality.
Back to the question at the beginning. What law does an AI crawler break?
Not complying with robots.txt, in itself, breaks no law.
But it is a notice posted at the door; every step you take after ignoring it may tread on something else: taking someone's business, picking someone's lock, and finally entering someone's system. What is pursued in judgments is never that line of Disallow, but what you did after crossing it.
In the data business, the four words "take it in the right way" now carry a price tag.
If you are building an AI product and have been blocked by a platform, or have received a lawyer's letter from the other side, you can leave a message in the backend to contact me.
Reference sources
• Civil Judgment No. 487 [2017] Jing Min Zhong of the Beijing High People's Court, second instance in the unfair competition dispute between Baidu Netcom and Baidu Online and Qihoo
• Civil Judgments No. 2668 [2013] Yi Zhong Min Chu Zi and No. 13657 [2013] Yi Zhong Min Chu Zi of the Beijing First Intermediate People's Court
• Civil Judgment No. 281 [2021] Jing Min Zhong of the Beijing High People's Court, second instance in the unfair competition dispute between Weimeng Chuangke and ByteDance
• Civil Judgment No. 2020 [2017] Jing 73 Min Chu of the Beijing Intellectual Property Court; Civil Judgment No. 24530 [2017] Jing 0108 Min Chu of the Beijing Haidian District People's Court
• Case notification issued by the Fuzhou Intermediate People's Court, "Publicly Available News Data Is Not a Free Resource: How Can Public-Opinion Analysis Crawl Data at Will?"; China Intellectual Property News report on the second-instance proceedings before the Fujian High People's Court
• "Profiting by Crawling and Using Navigation Electronic Map Data: Court of First Instance Finds Unfair Competition and Awards RMB 12.5 Million in Damages," published by the Chaoyang Court WeChat account of the Beijing Chaoyang District People's Court; affirmed on appeal by the Beijing Intellectual Property Court
• Anti-Unfair Competition Law of the People's Republic of China (second amendment adopted on June 27, 2025, effective October 15, 2025)
• Article 285 of the Criminal Law of the People's Republic of China; Interpretation of the Supreme People's Court and the Supreme People's Procuratorate on Several Issues Concerning the Application of Law in the Handling of Criminal Cases Endangering the Security of Computer Information Systems (Fa Shi [2011] No. 19)
• Article 49 of the Copyright Law of the People's Republic of China (as amended in 2020)
• Self-Discipline Convention on Internet Search Engine Services (Internet Society of China, November 1, 2012)
• Guiding Case No. 36 of the Supreme People's Procuratorate: Wei et al. Case of Illegally Obtaining Computer Information System Data
• EdgeOne, "AI Crawler Handling Practices: How E-Commerce Websites Can Prevent Product Data from Being Crawled by AI Crawlers" (vendor observation, not official statistics)
/ END.
*This article is an original article ofMankun Law Firmand represents only the personal views of the author. It does not constitute legal consultation or legal advice on any specific matter. For reprinting and legal consultation, please contact our customer service:mankunlawyer。
