Skip to content
CiteScore

Crawlers

AI crawler user agents: GPTBot, ClaudeBot, PerplexityBot, and the rest

A reference table of AI crawler and assistant user-agent tokens, with operator, purpose, whether public docs say they honor robots.txt, and a copyable robots.txt group. Unconfirmed cells say so.

18 October 2026 · CiteScore Team · 14 min read

Each row is a user-agent token an operator has published, or a token people write when the operator has not published one. The purpose and the robots.txt cell are limited to what that operator’s public docs say. Operator pages linked from the table were read on 18 October 2026. A vendor can change a token without notice.

The robots.txt checker sends you to the free check, which reads GPTBot, PerplexityBot, and Google-Extended on the live file. The robots.txt generator writes allow or block groups for OAI-SearchBot, ChatGPT-User, GPTBot, PerplexityBot, ClaudeBot, and Google-Extended. The GEO audit scores that same three-token crawler dimension. It does not log into ChatGPT, Claude, Perplexity, or Google.

The tokens

A named group does not inherit User-agent: *. If you disallow a private path for everyone, repeat that path inside the named group. The snippet is one group, not a complete file. Merge it with the Sitemap line and the Googlebot rules you already have. Replacing the whole file can drop a rule you meant to keep. robots.txt user-agent matching is case-insensitive. The snippet uses the spelling the operator publishes.

TokenOperatorPurposerobots.txtSnippet
GPTBotOpenAITraining. OpenAI says GPTBot crawls content that may be used to train generative AI foundation models. Disallowing it means that content should not be used for that training. The setting is independent of OAI-SearchBot.OpenAI documents a robots.txt tag for GPTBot. OpenAI’s advertiser guidance says OpenAI crawlers respect robots.txt and stop when access is disallowed. The bots overview separately says ChatGPT-User may not follow those rules.
User-agent: GPTBot
Disallow: /

Opt-out of training. It does not set the OAI-SearchBot rule.

OAI-SearchBotOpenAISearch. OpenAI says OAI-SearchBot surfaces sites in ChatGPT search. A site opted out of this token is not shown in ChatGPT search answers, though it can still appear as a navigational link.OpenAI documents this as its own robots.txt tag and recommends allowing it, plus the published IP ranges, if you want the site in search results. OpenAI says a robots.txt change can take about 24 hours to affect search.
User-agent: OAI-SearchBot
Allow: /

Allow, which is what OpenAI recommends for ChatGPT search. Disallow: / is the opt-out.

ChatGPT-UserOpenAIUser-triggered fetch. OpenAI says ChatGPT-User visits a page for certain actions a person starts in ChatGPT or a Custom GPT, including GPT Actions. It is not an automatic crawl, and it is not the control for Search. Use OAI-SearchBot for that.OpenAI says robots.txt rules may not apply, because a person started the action. Publishing a group is still possible. OpenAI does not say the group will be honored.
User-agent: ChatGPT-User
Disallow: /

A request only. OpenAI says robots.txt may not apply.

ClaudeBotAnthropicTraining. Anthropic says ClaudeBot collects web content that could contribute to training. Restricting it signals that the site’s future material should be excluded from training datasets.Anthropic says its bots honor industry-standard robots.txt directives, including Crawl-delay where appropriate. A disallow has to be repeated for each bot and each subdomain. A file on www does not cover the bare host.
User-agent: ClaudeBot
Disallow: /

Anthropic’s published shape for a full-site block. It does not write Claude-User or Claude-SearchBot.

Claude-UserAnthropicUser-triggered fetch. Anthropic says Claude-User accesses a site when a person asks Claude a question. Disabling it prevents retrieval in response to that query, which Anthropic says may reduce visibility in user-directed web search.Anthropic includes Claude-User in the bots that honor robots.txt. That is different from ChatGPT-User and Perplexity-User, where the operator says the user-triggered fetch may ignore the file.
User-agent: Claude-User
Disallow: /

Full-site block. Repeat it on every subdomain you want to opt out.

Claude-SearchBotAnthropicSearch. Anthropic says Claude-SearchBot navigates the web to improve search-result quality. Disabling it prevents indexing for that search use, which Anthropic says may reduce visibility and accuracy in search results.Same robots.txt statement as ClaudeBot: Anthropic’s bots honor standard directives. A ClaudeBot group does not write this token.
User-agent: Claude-SearchBot
Disallow: /

Full-site block for the search token, separate from ClaudeBot.

PerplexityBotPerplexitySearch. Perplexity says PerplexityBot surfaces and links sites in Perplexity search results. It is not used to crawl content for AI foundation models.Perplexity’s crawler docs tell you to allow this token in robots.txt, and the published IP ranges, if you want the site in search results. The setting is independent of Perplexity-User. Their help center says a disallow stops indexing of full or partial text. See the note under this table.
User-agent: PerplexityBot
Allow: /

Allow, which is what Perplexity recommends for search. Disallow: / is the opt-out.

Perplexity-UserPerplexityUser-triggered fetch. Perplexity says Perplexity-User visits a page when a person asks a question, so the answer can include a link. It is not used for web crawling or to collect content for training foundation models.Perplexity says this fetcher generally ignores robots.txt, because a person requested the fetch. A group does not replace a firewall rule.
User-agent: Perplexity-User
Disallow: /

A request only. Perplexity says this fetcher generally ignores robots.txt.

Google-ExtendedGoogleTraining and grounding control, not a separate crawl. Google says the token manages whether content Google has crawled may train future Gemini models that power Gemini Apps and Vertex AI, and whether it may be used for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It does not change inclusion or ranking in Google Search.Google documents it as a robots.txt control token. It has no separate HTTP user-agent, so it will not appear in an access log. The fetch is done by existing Google crawlers, which Google says obey robots.txt on automatic crawls.
User-agent: Google-Extended
Disallow: /

Full opt-out of that Gemini use. Google’s own example is a path rule, not a requirement to block the host.

Applebot-ExtendedAppleTraining control, not a crawler. Apple says publishers can opt out of content being used to train Apple’s foundation models for Apple Intelligence, Services, and Developer Tools. Pages that disallow it can still appear in Apple search. The crawl itself is Applebot.Apple documents Applebot-Extended as a robots.txt token that decides how data Applebot already crawled may be used. Applebot-Extended does not fetch pages. Apple says Applebot respects standard robots.txt directives. Apple’s published example disallows a path, not necessarily the whole host.
User-agent: Applebot-Extended
Disallow: /

Full opt-out of foundation-model training. It does not block Applebot search.

BingbotMicrosoftSearch index. Microsoft says Bingbot is Bing’s standard crawler and finds pages for the Bing index. Microsoft does not document Bingbot on that page as a separate token for generative-model training. The HTTP user-agent string uses bingbot.Microsoft’s crawler overview says a robots.txt file tells Bing crawlers how to interact with the site. Bing’s page on reporting a Bingbot issue says Bingbot honors robots.txt, including crawl-delay. Bing’s robots.txt guide says a Bingbot group ignores the * group, so general rules have to be repeated, and a page that is not crawled is not indexed. Disallow: / would ask Bing not to crawl the host.
User-agent: Bingbot
Disallow: /private/

A private-path example, not a suggestion to block Bing. Repeat any * rules inside this group.

CCBotCommon CrawlOpen web archive. Common Crawl says CCBot builds a public crawl dataset anyone can analyze. The CCBot page does not describe it as a search index, a user-triggered fetch, or a model Common Crawl trains. The HTTP user-agent they publish is CCBot/2.0.Common Crawl publishes a CCBot group with Disallow: / as the way to stop the crawl. The page does not say that block deletes a copy already in an older crawl.
User-agent: CCBot
Disallow: /

Common Crawl’s published full-site block.

Meta-ExternalAgentMetaTraining or indexing. Meta says Meta-ExternalAgent crawls for uses such as training foundation models or improving products by indexing content directly. The user-agent string Meta publishes is meta-externalagent, lowercase.Meta says to block a crawler by adding a disallow in robots.txt. Meta says Meta-ExternalFetcher may bypass robots.txt because a person requested the fetch, and that FacebookExternalHit might bypass it for security checks. Meta does not say Meta-ExternalAgent bypasses the file. Changes can take up to 24 hours.
User-agent: meta-externalagent
Disallow: /

Full-site block, using Meta’s documented token spelling. Meta’s example allows / and disallows /private/.

BytespiderByteDanceNot confirmed. The token is widely listed by third parties as ByteDance’s. On 18 October 2026, ByteDance’s Toutiao webmaster site returned a page whose HTML contained no Bytespider policy. This row does not assign a purpose.Not documented by an operator page we could read. Treat the snippet as a request, then check your own logs. There is no published commitment here that Bytespider honors it.
User-agent: Bytespider
Disallow: /

A request, not a confirmed control.

AmazonbotAmazonMay be used for training. Amazon says Amazonbot improves products and services and may be used to train Amazon AI models. Search is a different token, Amzn-SearchBot. A user-triggered fetch is Amzn-User. Those two are not this row.Amazon says automated crawling by the listed agents honors robots.txt allow and disallow, at the host level. Amazon says Amzn-User may not follow all robots.txt directives. That exception is not written for Amazonbot. They do not support crawl-delay.
User-agent: Amazonbot
Disallow: /

Full-site opt-out of Amazonbot. A path disallow is also valid. It does not write Amzn-SearchBot or Amzn-User.

DuckAssistBotDuckDuckGoAI answers, not training. DuckDuckGo says DuckAssistBot crawls pages in real time for AI-assisted answers that cite sources, and that this data is not used to train AI models. Opting out does not change organic rankings or whether the site appears in DuckDuckGo search results.DuckDuckGo says disallowing the user agent stops DuckAssistBot from crawling. The change takes effect after 72 hours.
User-agent: DuckAssistBot
Disallow: /

DuckDuckGo’s published opt-out. It does not block DuckDuckBot search.

MistralAI-UserMistralUser-triggered fetch. Mistral says MistralAI-User visits a page when a person asks Vibe a question, and may include a link to the source. It is not an automatic crawl and is not used to crawl content for generative AI training. Indexing is MistralAI-Index. Training is MistralAI-Training.Mistral documents a robots.txt tag and says MistralAI-User governs which sites those user requests can reach. Mistral does not say, as OpenAI and Perplexity do for their user fetchers, that this token ignores robots.txt.
User-agent: MistralAI-User
Disallow: /

The robots.txt tag Mistral documents for this user fetcher. It does not write MistralAI-Index or MistralAI-Training.

cohere-aiCohereNot documented under this name. Cohere’s crawler page says they do not currently use bots to crawl the web to train generative AI foundation models. The bot table on that page is empty. The example token they publish for a possible future crawler is Coherebot, not cohere-ai.Cohere says its policy is that crawlers respect robots.txt. They do not document a live crawler named cohere-ai, so this page cannot say that token honors the file.
User-agent: cohere-ai
Disallow: /

People write this token. Cohere’s own example is User-agent: Coherebot and Disallow: /. Cohere does not list either as a live crawler.

What the docs do not settle

Perplexity’s crawler documentation is the source for the PerplexityBot and Perplexity-User rows. Their help center article, “How does Perplexity follow robots.txt?”, adds a detail the crawler page does not: PerplexityBot will not index the full or partial text of a page disallowed in robots.txt, and may still index the domain, the title, and a short factual summary. The English URL of that article returned an access challenge on 18 October 2026. The sentence is from the Spanish and Chinese versions of the same article, which say they were last modified on 14 September 2026. If those versions and the English page ever disagree, use the English page.

Bytespider is included because server logs and third-party lists use the token. ByteDance’s public webmaster site did not return a readable crawler policy in the HTML, so the purpose and the robots.txt promise are blank on purpose. cohere-ai is included for the same reason. Cohere’s crawler page says they do not currently crawl to train foundation models, lists no live bot, and the example they publish is Coherebot. Writing cohere-ai into robots.txt is a request to a token Cohere has not documented.

Bingbot is Microsoft’s search crawler. This table does not call it a generative-model training token, because Microsoft’s crawler docs do not. Google-Extended and Applebot-Extended are control tokens. They do not show up as their own user-agent in an access log. Meta’s user-triggered fetcher is Meta-ExternalFetcher, which Meta says may bypass robots.txt. It is a different token from Meta-ExternalAgent and is not a row above. The token Meta’s crawler page ties to citations in Meta AI responses is Meta-WebIndexer, also not a row above. The longer note, including why this page does not treat FacebookBot as that control, is how to get cited by Meta AI. DeepSeek is not a row in the table. The Search and Fetch guide does not name a user-agent, and this page does not add one from a third-party list. The note is how to get cited by DeepSeek. Mistral’s crawler page names MistralAI-User, MistralAI-Index, and MistralAI-Training. The note is how to get cited by Mistral. Brave’s crawler page does not advertise a differentiated user agent. It says a page Googlebot cannot crawl is a page Brave will not crawl. This table does not add a Brave token. The note is how to get cited by Brave Search. You.com’s Web Search API page and Answer API guide do not name a robots.txt token. This table does not add one. The note is how to get cited by You.com. DuckAssistBot is the token DuckDuckGo publishes for AI-assisted answers. DuckDuckBot is the search crawler. The note is how to get cited by DuckDuckGo. Comet is Perplexity’s browser assistant. Perplexity’s crawler page does not name a Comet user agent, and this table does not add one. The note is how to get cited by Perplexity Comet.

Questions about the tokens

If User-agent: * allows the site, are these tokens allowed?

Only when that token has no group of its own. A crawler that honors robots.txt uses the group that names it, and uses User-agent: * when it has no group. An Allow on * does not override a Disallow written under the token. Several user-triggered fetchers are a separate case: OpenAI says ChatGPT-User may not follow robots.txt, and Perplexity says Perplexity-User generally ignores it.

Does blocking GPTBot remove a site from ChatGPT search?

No. OpenAI documents GPTBot and OAI-SearchBot as independent. GPTBot is the crawl that may be used to train generative models. OAI-SearchBot is the search crawler. A site opted out of OAI-SearchBot is not shown in ChatGPT search answers, though it can still appear as a navigational link. The same split exists elsewhere: ClaudeBot does not write Claude-SearchBot or Claude-User, and PerplexityBot does not write Perplexity-User.

Which of these tokens does the free check read?

The free check reads GPTBot, PerplexityBot, and Google-Extended. The robots.txt generator writes OAI-SearchBot, ChatGPT-User, GPTBot, PerplexityBot, ClaudeBot, and Google-Extended. This table is wider than both. A token that is not in the check can still be allow, disallow, or unspecified in your file.

What should I do when the operator has not documented the token?

Say so, and do not treat a third-party list as the operator’s promise. Bytespider and cohere-ai are in that state on this page. You can still publish a robots.txt group. Check your logs afterward. robots.txt is a request a well-behaved crawler can honor. It is not a firewall.

Get a free AI Answer Snapshot · Fix Pack, $49 · Run the free check

A separate GEO audit PDF covers one public URL.