Cloudflare declares war on AI companies, first blood was drawn on July 1st

One of the most popular Content Delivery Networks (CDN) available, Cloudflare, has recently announced it is drawing a line in the sand with AI companies.

2

The new initiative began on July 1 with Cloudflare announcing via a new blog post that it will be blocking AI web crawlers on the websites it hosts by default, preventing AI companies such as OpenAI, Anthropic, and others from scraping the websites' content. For those who are out of the loop, AI companies are continuously gathering new data through web crawlers that essentially act as leeches on websites, and in many instances, slowing down the speed of the website through their continuous requests.

However, Cloudflare is putting a stop to that, and as of July 1, all AI web crawlers will be blocked by default. This isn't a small change either, as Cloudflare has more than two million customers, which is approximately 20% of the web. Moreover, Cloudflare states it also has the capabilities to detect "shadow" scrapers, which are bots that attempt to evade detection by using various unpopular or sophisticated tactics.

So, what happens to AI vendors? AI companies now must request access and clarify their intentions when attempting to crawl a Cloudflare-hosted website. The request is then presented to the website owner, who makes the decision.

Notably, Cloudflare isn't just introducing a block for AI crawlers; it is also launching the "Pay Per Crawl" program, which enables website publishers to set their own rates for AI companies to buy the content being published on the website.

"Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure," writes Cloudflare