robots.txt

Free robots.txt AI crawler generator

Free, no-signup robots.txt generator for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Extended. Choose allow or block, then copy the file.

10 October 2026 · 5 min read

Starting point

Allow every crawler listedAllow AI search, block training crawlersBlock every crawler listed

OAI-SearchBot

ChatGPT search. OpenAI says a site that disallows this token is not shown in ChatGPT search answers.

ChatGPT-User

A fetch started by a person in ChatGPT. OpenAI says robots.txt may not apply, because a person started it.

GPTBot

OpenAI training crawl. Independent of OAI-SearchBot. Blocking it does not block ChatGPT search.

PerplexityBot

Perplexity’s crawler. Perplexity-User is a different token and is not listed here.

ClaudeBot

Anthropic’s crawler. Claude-SearchBot is a different token and is not listed here.

Google-Extended

Gemini Apps and Vertex AI training. Google says this token does not change Google Search or AI Overviews.

Private pathsOne path per line. Repeated on every Allow group, because a named crawler ignores User-agent: *.Sitemap URLOptional. A bare domain becomes /sitemap.xml.

robots.txt

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /

Generated in your browser. Nothing is uploaded.

If you already have a robots.txt, merge these named groups into it. Replacing the file can drop a Googlebot rule you meant to keep.

Run the free check · Get the $39 GEO audit

Which token does which job

A crawler uses the group that names it and ignores User-agent: *, so a private path has to be repeated on every Allow group. The generator does that. Unnamed agents, including Googlebot, follow the * group.

OAI-SearchBot — ChatGPT search. OpenAI says disallowing it keeps a site out of ChatGPT search answers.
ChatGPT-User — a fetch started by a person in ChatGPT. OpenAI says robots.txt may not apply.
GPTBot — OpenAI training. Independent of OAI-SearchBot.
PerplexityBot — Perplexity’s crawler. Perplexity-User is not listed; it follows *.
ClaudeBot — Anthropic. Claude-SearchBot is not listed; it follows *.
Google-Extended — Gemini Apps and Vertex AI training. Google says it does not change inclusion or ranking in Google Search, so it is not an AI Overviews switch.

How to publish it

If you already have a robots.txt, copy the named groups in and merge the private paths with the rules you already wrote for Googlebot. Replacing the whole file can drop a Sitemap line or a Googlebot group you meant to keep. If you do not have a file, the copy is a complete starter.

The free check reads the live file for GPTBot, PerplexityBot and Google-Extended. The $39 audit scores the page those crawlers would fetch. The longer notes on each token are on the crawler guide.

Run the free check · Get the $39 GEO audit