Reference

Which AI crawlers to block, and which you cannot afford to

Most published blocklists treat every AI crawler as the same thing and tell you to block the lot. For a site that sells, that advice is expensive: three of these tokens gate model training, five gate the answer indexes shoppers actually buy from, and four are fetches a customer personally asked for.

Four kinds of access, one file

robots.txt has one verb for all of them, which is why so much advice about it is wrong for stores. The distinction that decides whether a block costs you money is not who operates the crawler but what the crawler feeds.

Training. The content goes into a model's weights. Blocking it has no effect on whether an assistant can find your store today. This is the category where blocking is cheap, and it is the only category most blocklists are really about.
Answer indexes. The content goes into a retrieval index that an assistant searches when someone asks it to find a product. Blocking these removes your store from that surface outright. For a store this is the expensive mistake, and it is usually made by accident.
User-triggered fetches. A person pasted your URL, or asked their assistant to check a price. There is no corpus and no index: it is one request, for one person, who is already looking at your store. Blocking it is closer to blocking a browser than to blocking a crawler. OpenAI, Perplexity, Amazon and Meta each document that robots.txt may not be applied to these at all, so the rule may also simply not hold.
Usage tokens. Not crawlers. Google-Extended and Applebot-Extended fetch nothing and appear in no log; they only decide how content another agent already fetched may be used. You cannot block them by user-agent string, because there is no string to block.

The twenty-one tokens, and what each one gates

These are the tokens our scanner checks, with the vendor, the category and what blocking it actually costs. Tokens are matched case-insensitively and must appear exactly as written.

Scroll sideways for the full row
Token	Operator	Kind	What it gates	Cost of blocking
`OAI-SearchBot`	OpenAI	Answer index	Indexing for ChatGPT search results.	Removes the store from ChatGPT search answers. Rarely what anyone intends.
`ChatGPT-User`	OpenAI	On-demand	Fetches made because a person asked ChatGPT to look at a page.	Blocks a shopper who asked an assistant to check your price or stock. OpenAI states it “is not used to determine whether content may appear in Search”, and that because the action is user-initiated, “robots.txt rules may not apply” — so the block costs you no surface and may not hold.
`GPTBot`	OpenAI	Training	Crawling for OpenAI model training.	No effect on ChatGPT search or browsing. This is the one you can block with the least consequence.
`Google-Extended`	Google	Usage token	Whether already-crawled content may be used to train Gemini models and to ground answers in Gemini apps and Vertex AI.	Per Google's documentation, none in Google Search: not inclusion, not ranking. It is a token, not a crawler, so it never appears in your logs.
`Claude-SearchBot`	Anthropic	Answer index	Indexing for Claude's search results.	Removes the store from answers Claude composes from its index.
`Claude-User`	Anthropic	On-demand	Fetches made because a person asked Claude to look at a page.	Blocks a shopper who asked an assistant to open your product page. It is Claude-SearchBot, not this, that decides whether the store is in Claude’s index.
`ClaudeBot`	Anthropic	Training	Crawling for Anthropic model training.	No effect on Claude's search or browsing surfaces, which use separate tokens.
`PerplexityBot`	Perplexity	Answer index	Indexing for Perplexity answers.	Removes the store from Perplexity's cited results.
`Perplexity-User`	Perplexity	On-demand	Fetches made on behalf of someone using Perplexity.	Blocks a shopper mid-question, and may not even do that: Perplexity states that “since a user requested the fetch, this fetcher generally ignores robots.txt rules”. Inclusion in Perplexity’s answers is PerplexityBot’s business.
`Applebot-Extended`	Apple	Usage token	Whether content already crawled by Applebot may train Apple's generative foundation models. It crawls nothing itself.	Per Apple's documentation, none in Apple's search surfaces: pages that disallow it can still appear in results, and its rules are not considered in ranking. Siri and Spotlight follow plain Applebot.
`Amzn-SearchBot`	Amazon	Answer index	Indexing for Amazon's search experiences. Amazon says it “does not crawl content for generative AI model training”.	Allowing it is what makes content “eligible to appear in search experiences such as Alexa”, so a block removes the store from an explicitly commercial surface. This is the Amazon token that matters, and it is the one most blocklists omit.
`Amzn-User`	Amazon	On-demand	Fetches made for a user action, such as an Alexa question needing current information.	Blocks a shopper asking Alexa about you. Amazon notes it “may not follow all robots.txt directives”, since a user can initiate it.
`Amazonbot`	Amazon	Training	Crawling to “improve our products and services”, which Amazon says “may be used to train Amazon AI models”.	Not the shopping surface, despite its name and despite how often it is listed as one. Amazon documents no removal outcome for blocking it; search eligibility rides on Amzn-SearchBot instead.
`Meta-WebIndexer`	Meta	Answer index	Crawling that “navigates the web to improve Meta AI search result quality for users”.	Meta says allowing it “helps us cite and link to your content in Meta AI’s responses”, so blocking it costs you the citation. This is Meta’s answer surface, not meta-externalagent.
`Meta-ExternalFetcher`	Meta	On-demand	Fetches of individual links at a user’s request.	Blocks a shopper mid-question. Meta notes it “may bypass robots.txt rules”.
`meta-externalagent`	Meta	Training	Crawling “for use cases such as training foundation AI models or improving products by indexing content directly”.	Meta states no answer-surface effect for this token, so we claim none. We previously called it Meta AI’s answer crawler; that was wrong, and Meta-WebIndexer is the token that does it.
`MistralAI-Index`	Mistral	Answer index	Automated crawling “for indexing purposes only”, to index content for Mistral search. Content it crawls “is not used for generative AI training of any kind”.	Removes the store from Mistral’s search, which is what answers questions in Vibe. The indexer, not MistralAI-User, is the one with a surface attached.
`MistralAI-User`	Mistral	On-demand	Fetches made because someone asked Vibe, the assistant formerly called Le Chat, a question.	Blocks a shopper using Mistral’s assistant. Mistral says it is not used for automatic crawling and not used for training.
`MistralAI-Training`	Mistral	Training	Crawling to build datasets for training Mistral models. Not used for search indexing or live queries.	No search effect. Mistral documents this as blockable through robots.txt.
`Bytespider`	ByteDance	Training	ByteDance crawling that feeds its model training. ByteDance publishes no crawler documentation we could reach, so this is the most any source supports.	Widely blocked for crawl volume rather than for policy. No operator documentation ties it to a shopping surface, and the third-party lists claiming one disagree with each other.
`CCBot`	Common Crawl	Training	Common Crawl, the open corpus many models are trained from.	Indirect and hard to measure: Common Crawl feeds third parties you will never hear from.

Two things worth stating plainly

Google-Extended is the one most often described wrongly, including in places that should know better. It is a robots.txt token rather than a crawler, so you will never see it in a server log and you cannot block it by user-agent string. What it gates is narrower than its reputation: whether already-crawled content may train future Gemini models and ground answers in Gemini apps and on Vertex AI. Google states in the same place that it “does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search”, and Google's separate guidance on AI features points to Googlebot's own robots.txt rules, not this token, as the control for AI Overviews and AI Mode. We read that on Google's crawler documentation directly rather than taking it from a published blocklist, which is why we state it flatly. We have still not observed the behaviour ourselves, and could not: the token leaves no trace to observe.

robots.txt is a request, not an access control. It is honoured by the operators above as a matter of policy, and user-triggered fetchers in particular are not consistently documented as consulting it. If you need something not to be fetched, the control belongs at your server, not in a text file.

The three mistakes that undo the file

Naming a crawler replaces the wildcard group for it. A crawler obeys exactly one group: the most specific one that matches its name. Adding a named group does not add rules on top of User-agent: * — it takes that crawler out of the wildcard group entirely.

robots.txta surprise, not a bug

User-agent: *
Disallow: /private/
Crawl-delay: 10

User-agent: GPTBot
Disallow: /

# GPTBot now obeys ONLY its own group. The Disallow: /private/ and the
# Crawl-delay above do not apply to it any more. Naming a crawler replaces
# the wildcard group for that crawler; it does not add to it.

Within a group, the longest matching rule wins. Not the first, and not the last. An Allow only overrides a Disallow when its path is at least as long.

robots.txtlongest match wins

User-agent: OAI-SearchBot
Disallow: /products/
Allow: /products/winter-coat

# /products/winter-coat is allowed: the longer rule wins, and when two rules
# are the same length Allow beats Disallow. This is the standard's
# longest-match rule, and it is what our scanner implements.

A CDN toggle can contradict the file. The common one-click “block AI bots” setting at a CDN or WAF operates on user agents and IP ranges, above your robots.txt. A store can have a perfectly permissive file and still return a challenge page to every crawler on this list, which is also why our scanner reports an intercepted check as unverified instead of scoring it. If a scan comes back unverified rather than failed, look there first.

A robots.txt for a store that wants to be bought from

This blocks training, keeps every answer surface open, treats customer-triggered fetches like a browser, and keeps all of them out of the pages that only make sense for a logged-in human. Adjust the blocked paths to your own platform.

robots.txtstarting point

# A store that wants to be bought from. Order does not matter to a crawler,
# but grouping by intent keeps the file reviewable by a human.

# Training, and reuse of what was already crawled. Costs you no shopping surface.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Answer and shopping surfaces. Blocking these removes the store from them.
# Note Amzn-SearchBot and Meta-WebIndexer, not Amazonbot and meta-externalagent:
# the vendors split indexing from training, and the index is the one that sells.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: Meta-WebIndexer
User-agent: MistralAI-Index
Disallow: /cart
Disallow: /checkout
Disallow: /account
Allow: /

# Fetches a shopper asked for. Treat these like a browser.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Amzn-User
User-agent: Meta-ExternalFetcher
Disallow: /cart
Disallow: /checkout
Disallow: /account
Allow: /

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://yourstore.com/sitemap.xml

Check it on your own store

The scan reads your robots.txt, applies the longest-match rule the standard defines, and tells you which of the tokens above are blocked and what each block costs. It also tells you if your own file is asking our scanner not to look, because we honour that.

Run the free scan

No account, no email. It reports what it measured and says so when it could not measure something.