Cloudflare CEO says the web's future is "pay to crawl" as bots overtake human traffic
Quick Answer
Cloudflare CEO Matthew Prince reports that bot traffic now exceeds human traffic on the internet, accelerated by AI agents.
Quick Take
He predicts a future where accessing web content will require payment, coining the term 'pay to crawl.'
Key Points
- Bot traffic has surpassed human traffic on the internet.
- Matthew Prince attributes the surge to AI agents.
- The prediction for the web's future is 'pay to crawl.'
- This shift could impact content accessibility for users.
- The trend is occurring years ahead of previous forecasts.
📖 Reader Mode
~1 min readBot traffic has overtaken human traffic on the internet, according to Cloudflare CEO Matthew Prince. Prince says he didn't expect this tipping point until late 2027, but rapid growth in automated traffic from AI agents moved the timeline up fast. The shift happened over the past few months. The data is still rough, but the trend is unmistakable, Prince adds.
Prince notes that bot, crawler, and agent all mean the same thing; the label just depends on whether you see them as good or bad. As for the future of the web, "clearly it's going to be pay to crawl," he writes.
Cloudflare launched a platform last summer that lets site owners gate AI crawlers and charge for access, but it hasn't gained traction. Prince says his company is still working on "protocols and infrastructure to support the volume it'll require." Meanwhile, Google's AI Overviews and AI Mode have racked up billions of users.
— Originally published at the-decoder.com
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from The Decoder
See more →
An AI model programmed nonstop for 19 days on a single MirrorCode task that cost $2,600 to run
Epoch AI's MirrorCode benchmark reveals Claude Opus 4.7 as the leader with a 56% solve rate, reconstructing a 16,000-line toolkit in 14 hours. Despite this, all models tested struggle with the most complex tasks, highlighting limitations in current AI capabilities. The single task consumed $2,600 over 19 days, raising questions about cost-effectiveness in AI development.
