Cloudflare is imposing new default rules starting September 15, 2026 that will block AI crawlers from accessing ad-supported websites unless site owners opt in, forcing AI companies to negotiate content access separately from search engines.
This shift addresses a long-standing concern among publishers: while they want their content discoverable via search and AI, they do not want to provide it free for AI training.
Cloudflare is also rolling out a "Pay Per Use" model that lets publishers charge AI companies when their content generates value.
What happened
Cloudflare will change its default settings on September 15, 2026 to block "mixed-use" crawlers—those that blend search, agent use, and training—from accessing pages that host ads. The change applies to new customers, new sites by existing customers, and all free customers.
Why it matters
Website owners want their content discoverable via search and AI services, but they also want protection against having their intellectual property used for free. Cloudflare notes that Google has access to roughly 2x more information than other AI companies because it ties search discoverability to AI access. The shift also comes as bot traffic has surpassed human traffic on the internet for the first time.
What to watch
Cloudflare is expanding its "Pay Per Crawl" marketplace into "Pay Per Use," allowing publishers to charge AI companies when their content creates value. The company is working initially with Ceramic.ai and You.com; other AI companies can customize the model for their own use cases.
Ask the AI about this article →
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
