On September 15, 2026, part of the ad-supported web starts turning bots away at the door. Cloudflare says it will block AI training and agent crawlers by default on pages that carry ads: automatic for new domains, opt-out for existing ones. This looks like the clearest signal yet that the first attempt to charge machines for content did not work.
That attempt is barely a year old. Pricing a crawl was a first draft, and the rewrite, how to pay for the value content creates rather than the traffic it absorbs, is being drafted in public right now.
The first draft metered the crawl
In July 2025 Cloudflare shipped Pay Per Crawl in private beta. A publisher sets a flat price per request, and each AI crawler is allowed through free, charged the fee, or blocked. It ran on HTTP 402, the payment status code we wrote about recently.
The plumbing worked. The pricing came from the search era, where charging for a fetch made sense because the fetch led somewhere: the page got indexed, and readers followed later. AI crawling drops that second half. Nobody comes back, and a fetch that leads nowhere is worth whatever the fee happens to say it is.
A crawl was never a unit of value
More than half of the crawl traffic from well-behaved bots re-fetches pages that have not changed since the last visit, Cloudflare says. The number is Cloudflare's, measured on its own network, so treat it as directional. A flat fee bills every redundant re-fetch at full price and charges nothing extra for the fetch whose content ends up quoted in a million answers. Price rises with volume. Value does not track volume at all.
Pay per use
Cloudflare's answer arrived on July 1, 2026, under the label Content Independence Day: pay publishers when their content creates value, when it shows up in an answer, instead of when a bot fetches it. That forces a publisher to specify two things pay-per-crawl never asked. What a machine may do with the page: Immediate reads and stores nothing, Reference (the default) allows indexing, excerpting, and linking back, Full allows summarizing and reproducing. And who is asking: search, agent actions, or training. A publisher can welcome a search crawler at Reference and refuse a training crawler outright.
The September block is the same idea applied to inaction. New domains get the new defaults automatically, existing owners are moved onto them unless they opt out first. Most sites never touch a default, which is why setting them is where the power sits.
Trading one gatekeeper for another
Pay-per-use also can't be enforced where pay-per-crawl could. A crawl is billable at the edge, on infrastructure the publisher or its CDN owns. Value in an answer is created downstream, inside a response nobody upstream ever sees, so collecting on it means trusting the AI company's accounting or seating a measurer inside the answer pipeline. The model needs a party in the middle.
Big publishers don't need one, they sign bilateral licensing deals. A recipe blog, a regional outlet, a documentation site, a forum with ten years of hard-won answers: none of them can. The only thing that reaches the long tail is a standard applied across millions of sites at once, and that requires a middleman with reach. Cloudflare has it, sitting in front of a large share of web traffic, on the order of a fifth by its own accounting, while designing the pricing model, setting the defaults, and offering to measure use and move the money.
So the publisher who could never bargain with Google is offered a way out that runs through Cloudflare instead. The dependence does not disappear. It changes address. A cut beats zero, and the incentives point the same way for now.
The question has moved either way. A year ago it was whether machines should pay to fetch content at all. Now it is who prices the value they extract, who collects, and who is trusted to measure a transaction that happens where no one can see it. If content is priced and gated for software buyers, the businesses that come out ahead are the ones that make themselves legible to those buyers on purpose: discoverable, quotable, ready to transact. That is a design problem, and where this series goes next.
This is Part 4 of The Machine Web, a series on what changes when the web's main character stops being human.
Part 1: The Machine Web Part 2: The reader isn't human anymore Part 3: HTTP 402 wakes up
AKENA is a blockchain engineering studio. We build the infrastructure AI agents run on: agent-facing RPC and MCP endpoints, on-chain data pipelines, and the products on top of them. If you're deciding how to price, gate, or sell to machine clients, we should talk.