Cloudflare CEO says the web's future is "pay to crawl" as bots overtake human traffic
Quick Answer
Cloudflare CEO Matthew Prince reports that bot traffic now exceeds human traffic on the internet, accelerated by AI agents.
Quick Take
He predicts a future where accessing web content will require payment, coining the term 'pay to crawl.'
Key Points
- Bot traffic has surpassed human traffic on the internet.
- Matthew Prince attributes the surge to AI agents.
- The prediction for the web's future is 'pay to crawl.'
- This shift could impact content accessibility for users.
- The trend is occurring years ahead of previous forecasts.
Source Excerpt
Bot traffic now outpaces human traffic on the internet, Cloudflare CEO Matthew Prince says, years ahead of his late 2027 forecast. He blames AI agents for the surge. His conclusion for the future of the web: "Clearly it's going to be pay to crawl. "
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from The Decoder
See more →
An AI model programmed nonstop for 19 days on a single MirrorCode task that cost $2,600 to run
Epoch AI's MirrorCode benchmark reveals Claude Opus 4.7 as the leader with a 56% solve rate, reconstructing a 16,000-line toolkit in 14 hours. Despite this, all models tested struggle with the most complex tasks, highlighting limitations in current AI capabilities. The single task consumed $2,600 over 19 days, raising questions about cost-effectiveness in AI development.
