When do these new blocking rules take effect?

Cloudflare's default settings will block mixed-use crawlers starting on September 15, 2026. The change applies to new customers, new sites set up by existing customers, and all existing free customers.

Can website owners still allow AI crawlers after September 2026?

Yes. Site owners can adjust Cloudflare's settings to allow mixed-use crawlers to access their pages. Cloudflare is also introducing "Pay Per Use," which allows publishers to charge AI companies when their content creates value, giving them a financial incentive to permit access.

Which AI companies is Cloudflare working with on the new payment model?

Cloudflare is initially working with Ceramic.ai and You.com. When a publisher opts in, they are paid when their content appears in Ceramic's AI search results or when You.com accesses their premium content. Other AI companies can customize this model for how they work.

Back to articles

Cloudflare blocks AI crawlers from ad-supported sites starting September 2026

TechCrunch AI13h ago5 min read

Key takeaway

Cloudflare is imposing new default rules starting September 15, 2026 that will block AI crawlers from accessing ad-supported websites unless site owners opt in, forcing AI companies to negotiate content access separately from search engines. This shift addresses a long-standing concern among publishers: while they want their content discoverable via search and AI, they do not want to provide it free for AI training. Cloudflare is also rolling out a "Pay Per Use" model that lets publishers charge AI companies when their content generates value.

Summaries like this, in your inbox every morning.

3 Key Points

What happened
Cloudflare will change its default settings on September 15, 2026 to block "mixed-use" crawlers—those that blend search, agent use, and training—from accessing pages that host ads. The change applies to new customers, new sites by existing customers, and all free customers.
Why it matters
Website owners want their content discoverable via search and AI services, but they also want protection against having their intellectual property used for free. Cloudflare notes that Google has access to roughly 2x more information than other AI companies because it ties search discoverability to AI access. The shift also comes as bot traffic has surpassed human traffic on the internet for the first time.
What to watch
Cloudflare is expanding its "Pay Per Crawl" marketplace into "Pay Per Use," allowing publishers to charge AI companies when their content creates value. The company is working initially with Ceramic.ai and You.com; other AI companies can customize the model for their own use cases.

FAQ

When do these new blocking rules take effect?: Cloudflare's default settings will block mixed-use crawlers starting on September 15, 2026. The change applies to new customers, new sites set up by existing customers, and all existing free customers.
Can website owners still allow AI crawlers after September 2026?: Yes. Site owners can adjust Cloudflare's settings to allow mixed-use crawlers to access their pages. Cloudflare is also introducing "Pay Per Use," which allows publishers to charge AI companies when their content creates value, giving them a financial incentive to permit access.
Which AI companies is Cloudflare working with on the new payment model?: Cloudflare is initially working with Ceramic.ai and You.com. When a publisher opts in, they are paid when their content appears in Ceramic's AI search results or when You.com accesses their premium content. Other AI companies can customize this model for how they work.

Discussion

No discussion yet for this article

Stay ahead with AI news

Get curated AI news from 200+ sources delivered daily to your inbox. Free to use.

Get Started Free

Free · takes 30 seconds · unsubscribe anytime

1 minute a day. The AI essentials.

200+ sources · Email / LINE / Slack

Get it free →