Block AI Crawlers: The 15 Bots That Matter

Most lists that claim to help you block AI crawlers are copy-pasted and dangerously wrong about the two tokens that actually matter. Sorting them properly is not an abstract taxonomy exercise. It is the single decision that determines whether your content vanishes from AI answers while training continues — or vice versa. We maintain the crawler registry in lib/ai-crawlers.ts that powers techpotions’ free AI robots.txt generator . Every agent string and description was verified against the operator’s own crawler documentation. The registry holds 15 verified bots across four categories, and that four-way split is this article’s structure, because the categories map directly to what blocking costs you. Two tokens almost everyone gets wrong Before the list, the single most important correction to make, and almost every listicle on this query gets it wrong: Google-Extended and Applebot-Extended are not crawlers. They are robots.txt tokens — product controls that govern whether your content is used for Gemini and Apple foundation-model training. Blocking Google-Extended does not affect Google Search crawling, Google ranking, or regular Applebot search indexing. People block them believing they are opting out of AI Overviews, and are actually opting out of nothing they think they are, while leaving search indexing completely untouched. Platforms have started wiring these tokens into one-click controls. Cloudflare’s managed robots feature, released mid-2025, lets you add AI crawler rules through a dashboard toggle rather than editing a raw file — but the underlying token logic above still applies. Block AI crawlers: the four categories that decide the cost Every AI crawler we track belongs to one of four categories. The category tells you the cost of blocking it. Training crawlers scrape pages to feed a model that may never cite you. Blocking them is a defensive data decision. Assistant crawlers fetch pages to answer a live user’s question and can cite and link you. Blockin