Redirects for AI Training enforces canonical content
Quick Answer
Cloudflare introduces Redirects for AI Training, enabling verified AI crawlers to be redirected to current content via 301 redirects, addressing issues with deprecated documentation being crawled.
Quick Take
This feature leverages existing canonical tags and aims to improve the accuracy of AI training data by ensuring outdated content is not ingested by AI models.
Key Points
- Redirects for AI Training automatically converts canonical tags to 301 redirects for verified AI crawlers.
- AI crawlers visited deprecated content 4.8 million times in the last 30 days, showing no respect for advisory signals.
- Cloudflare's solution targets bots like GPTBot, ClaudeBot, and Bytespider specifically.
- The feature is available on all paid Cloudflare plans with a simple toggle.
- Redirects do not affect human traffic or unverified crawlers, focusing solely on AI training bots.
Source Excerpt
Soft directives don’t stop crawlers from ingesting deprecated content. Redirects for AI Training allows anybody on Cloudflare to redirect verified crawlers to canonical pages with one toggle and no origin changes.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from Cloudflare AI
See more →
Temporary Cloudflare Accounts for AI agents
Cloudflare introduces Temporary Accounts on Cloudflare Workers, enabling AI agents to deploy live Workers instantly using 'wrangler deploy --temporary'. This feature removes barriers for AI agents, allowing them to operate more efficiently in real-time environments.
