The internet runs on a simple deal. Websites share their content openly, and in return, platforms that use that content send traffic back. It has worked this way for decades. AI and the open web are now testing whether that deal can survive.

New data from Cloudflare, which powers roughly 20% of the internet, offers one of the clearest pictures yet of how that deal is breaking down. The company tracks how often AI bots crawl websites versus how often those platforms send users back through referrals. The gap between the two numbers is striking.

Anthropic’s bots crawl webpages 8,800 times for every single referral they send back. OpenAI’s ratio sits at 993 to 1. For context, Google’s ratio is 5.6 to 1. DuckDuckGo’s is 1.5 to 1.

The web built AI. The question now is whether AI will return the favor.

The Deal That Built the Internet

For most of the web’s history, the arrangement between content creators and platforms was implicit but functional. Websites allowed search engines to crawl and index their content for free. In return, those search engines sent users back to the original sources. Traffic meant ad revenue, subscriptions, and visibility. Everyone had something to gain.

AI and the open web don’t work that way. When someone asks a chatbot a question, the chatbot answers directly. There is no reason to click through to the original source. The information gets used, but the website that produced it sees nothing in return — no traffic, no revenue, no acknowledgment.

Moreover, the crawling itself has costs. AI bots don’t just visit a page and move on. They hit sites repeatedly, in large volumes, sometimes flooding servers with millions of requests over a matter of days. That activity drives up infrastructure costs for site owners without delivering any of the benefits that traditionally justified allowing crawlers in.

“What’s better than using people’s content for free?” is not a rhetorical question anymore. For AI companies right now, the honest answer is nothing. The current setup is almost entirely in their favor. Furthermore, as more users turn to chatbots for answers instead of clicking through to websites, the economic foundation that has kept the open web running starts to crack.

Who Pays the Price

The people most affected by this shift are not the big platforms. They are the writers, journalists, researchers, and independent publishers who produce the content that AI models are trained on and built around.

A news outlet that spends money reporting a story used to earn traffic when search engines indexed it. That traffic paid for the next story. Today, a chatbot can summarize that story in three sentences and send nobody back to the original source. The outlet absorbs the cost of the reporting. The AI company absorbs the value.

This is not a hypothetical. Already, 79% of major news publishers block AI training bots through their robots.txt files, according to data from BuzzStream. That number is growing. Publishers are not doing this out of stubbornness. They are doing it because the current arrangement gives them no reason to participate.

The irony in Anthropic’s case is particularly sharp. The company has built its reputation on being the most careful, most ethical player in AI. Its crawl-to-refer ratio of 8,800 to 1 suggests that reputation has not yet extended to how it treats the web that feeds it. Anthropic has previously questioned Cloudflare’s methodology, and pointed to growing referral traffic from newer features. However, the broader pattern across the industry is hard to dismiss.

What Comes After the Free Lunch

The current situation is not stable. AI and the open web are heading toward a conflict that neither side can fully afford to lose.

If publishers keep blocking AI crawlers, the quality of AI outputs degrades over time. Models trained on a shrinking, increasingly paywalled web will reflect that narrowing. On the other side, if the web’s economic engine keeps losing traffic to chatbots without compensation, the incentive to produce original verified content collapses. AI ends up feeding on a web that has less and less worth feeding on.

Cloudflare is trying to address this with a new marketplace where publishers can license their content to AI companies directly. It is a reasonable idea. Whether it gains traction is another question entirely. After all, the current arrangement costs AI companies nothing. Moving to a paid model requires them to give something up voluntarily.

Some pressure is coming from outside the industry. Legal challenges around training data and copyright are working their way through courts in several countries. Regulation is slow, but it is moving.

For now, the most honest thing to say about AI and the open web is this. The web made AI possible. The content that millions of writers, journalists, and researchers produced over decades became the raw material for some of the most valuable technology ever built. What the web gets back for that contribution, so far, is very little. That might change. But it will not change on its own.