Bot traffic surpasses human traffic on web; Cloudflare CEO backs "pay to crawl" model

What happened

Bot traffic has overtaken human traffic on the internet, accounting for 57.4 percent of HTTP requests worldwide versus 42.6 percent from humans, according to Cloudflare CEO Matthew Prince. The shift occurred over the past few months, driven by rapid growth in automated traffic from AI agents—sooner than Prince expected.

Why it matters

The reversal reflects how AI systems are now consuming vast amounts of web data. Cloudflare launched a platform last summer that lets site owners gate AI crawlers and charge for access, but it has not gained traction. Prince argues the web's future will require a "pay to crawl" model supported by new protocols and infrastructure, whereas currently Google's AI Overviews and AI Mode have racked up billions of users with no clear monetization for content creators.

What to watch

Prince notes the data is still rough, but the trend is unmistakable. The company is still working on the protocols and infrastructure needed to support the volume a paid-crawl model would require.