CiteScore

robots.txt

AI crawler robots.txt checker

Check whether your robots.txt allows or blocks GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended, with copy-paste rules for each.

9 October 2026 · 6 min read

Check your robots.txt in seconds

Paste your domain into the free AI visibility check. It reads /robots.txt and reports whether GPTBot, PerplexityBot and Google-Extended are allowed, blocked, or falling back to the * group.

The AI user agents that matter

  • OAI-SearchBot – ChatGPT search results.
  • GPTBot – OpenAI training crawl.
  • ChatGPT-User – fetches when a user asks ChatGPT to open a link.
  • PerplexityBot – Perplexity's index.
  • ClaudeBot – Anthropic.
  • Google-Extended – controls Gemini training use; it does not remove you from AI Overviews.

Allow answer engines, keep a private area private

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

Common mistakes

Blanket Disallow: / left over from staging; a WAF that 403s bots even when robots.txt allows them; and assuming blocking Google-Extended hides you from AI Overviews. See our crawler guide for details.

Run the free check