Cloudflare replaces its blanket AI bot block with granular controls for search, training, and agent crawlers
Quick Answer
Cloudflare introduces granular controls for AI bot access, allowing customers to manage Search, Training, and Agent bots separately.
Quick Take
Starting September 2026, Training and Agent bots will be blocked by default on ad-supported pages, while a new BotBase database provides enterprises with detailed bot classifications and access rules.
Key Points
- Customers can now block AI bots by category: Search, Training, and Agent.
- Training and Agent bots will be blocked by default on ad-supported pages from September 2026.
- Multi-purpose crawlers like Googlebot will follow the strictest access rule.
- Cloudflare's new BotBase database offers detailed insights into bot classifications.
- Verified bots must now prove honest identification to gain access.
📖 Reader Mode
~2 min readCloudflare gives all customers new tools to control how AI bots access their websites.
Since July 2024, Cloudflare customers have been able to block all AI crawlers with a single click. Now, instead of a blanket block, site owners can choose between three categories. Search covers search engine indexing. Training covers data collection for AI model training. Agent covers bots acting on behalf of users, like ChatGPT.
Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that carry ads, since advertising signals that a site wants human visitors. Search crawlers will still be allowed. Multi-purpose crawlers like Googlebot that both search and train will be treated according to whichever rule is strictest. The new options are available to users on the free plan as well.
Cloudflare is pushing bot operators to split their crawlers by purpose. Cloudflare CEO Matthew Prince has previously called out Google for bundling its search crawler with its AI crawler. In June 2026, Prince noted that bot traffic had surpassed human traffic on the internet for the first time, a milestone he originally didn't expect until late 2027.
New bot database gives enterprises more granular control
For enterprise customers, Cloudflare is also launching BotBase, a searchable database of all known bots built right into the dashboard. It shows how each bot is classified and how it uses content, whether it just links to pages or reproduces them in full.
Cloudflare is also changing the rules for verified bots. Previously, all verified bots were allowed through automatically. Going forward, a bot's category determines whether it gets access. The "Verified" label alone won't be enough anymore: Bot operators will need to prove they identify themselves honestly and don't abuse the access they're granted.
— Originally published at the-decoder.com
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from The Decoder
See more →
An AI model programmed nonstop for 19 days on a single MirrorCode task that cost $2,600 to run
Epoch AI's MirrorCode benchmark reveals Claude Opus 4.7 as the leader with a 56% solve rate, reconstructing a 16,000-line toolkit in 14 hours. Despite this, all models tested struggle with the most complex tasks, highlighting limitations in current AI capabilities. The single task consumed $2,600 over 19 days, raising questions about cost-effectiveness in AI development.
