Free tool · no signup · reads robots.txt only
AI Crawler Checker — Which AI Bots Can Reach You, and What Each Block Costs
Blocking GPTBot does not remove you from ChatGPT Search. That job belongs to
OAI-SearchBot, a different user-agent with its own robots.txt rule — and the same split
runs through ClaudeBot vs Claude-SearchBot and
PerplexityBot vs Perplexity-User. Paste a URL and this checker reads the
site's robots.txt, then reports all 23 tokens grouped by the job each one does, with the exact line that
matched. Opting out of training and staying in AI answers is possible, but only if each bot gets its own
User-agent group.
Fetches /robots.txt once and parses it. Nothing is stored, no request is sent as any bot, and a rule the site serves only to real crawler IPs will not appear here.
What this AI crawler checker checks
The tool inspects the one file that governs crawler access — your robots.txt — and reports it precisely. It does not send a live request as each bot, so a result reads “Allowed by robots.txt”, never “AI can definitely reach your site”.
- robots.txt reachability and HTTP status
- Crawler-specific
User-agentgroups - The most specific rule per bot (Allow / Disallow, longest-match)
- Blanket
User-agent: *rules that catch AI bots with no rule of their own - Sitemap declarations in robots.txt
- Live per-bot user-agent request (planned)
- Meta robots / X-Robots-Tag headers (planned)
- Cloudflare / WAF bot rules and 403s (planned)
- llms.txt structure (now live — llms.txt Validator)
A site can pass this robots.txt check and still block AI bots at the firewall. If you suspect a WAF block, verify separately.
Supported AI crawlers and what blocking each one affects
Twenty-three tokens, grouped by the job the vendor documents for each. The strings are copied from vendor documentation rather than from other people's blocklists, which is where most of the misspelled agents in the wild come from — a rule written for GPTbot or Perplexity-Bot matches nothing at all.
| Crawler | Company | Category | What blocking may affect |
|---|---|---|---|
| AI search / discovery | |||
| OAI-SearchBot | OpenAI | AI search / discovery | Eligibility to appear as a linked source in ChatGPT Search. |
| PerplexityBot | Perplexity | AI search / discovery | Eligibility to appear as a Perplexity answer source. |
| Claude-SearchBot | Anthropic | AI search / discovery | Eligibility to be cited by Claude web search. |
| DuckAssistBot | DuckDuckGo | AI search / discovery | Inclusion in DuckAssist answers. |
| YouBot | You.com | AI search / discovery | Inclusion in You.com AI answers. |
| User-triggered fetch | |||
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT’s ability to open your page on a user’s request. |
| Claude-User | Anthropic | User-triggered fetch | Claude’s ability to open your page on a user’s request. |
| Perplexity-User | Perplexity | User-triggered fetch | Perplexity opening your page on a user’s request. |
| Meta-ExternalFetcher | Meta | User-triggered fetch | Meta AI opening a link a user shared. |
| Model training | |||
| GPTBot | OpenAI | Model training | Whether your content is used to train OpenAI models. |
| Google-Extended | Model training | Whether your content trains Gemini — not Google Search or AI Overviews. | |
| ClaudeBot | Anthropic | Model training | Anthropic’s general crawl of your site. |
| anthropic-ailegacy | Anthropic | Model training | Legacy Anthropic crawling (mostly historical). |
| CCBot | Common Crawl | Model training | Inclusion in the Common Crawl dataset used by many models. |
| Google-CloudVertexBot | Model training | Vertex AI customer agents fetching your pages. | |
| Applebot-Extended | Apple | Model training | Whether your content trains Apple Intelligence. |
| Meta-ExternalAgent | Meta | Model training | Whether your content trains Meta’s models. |
| Bytespider | ByteDance | Model training | ByteDance model training and crawl load. |
| cohere-ai | Cohere | Model training | Whether your content trains Cohere models. |
| Traditional search | |||
| Googlebot | Traditional search | Google Search AND AI Overviews together — rarely advisable. | |
| Bingbot | Microsoft | Traditional search | Bing, Copilot, and indirectly ChatGPT Search. |
| Applebot | Apple | Traditional search | Siri and Spotlight suggestions. |
| Amazonbot | Amazon | Traditional search | Alexa answers and Amazon indexing. |
AI search crawlers vs AI training crawlers
What is the difference between an AI search crawler and an AI training crawler? An AI search crawler — such as OAI-SearchBot, PerplexityBot or Claude-SearchBot — indexes your pages so an AI engine can find, quote and link them when answering a question. These are the bots that make you eligible to be cited. An AI training crawler — such as GPTBot, Google-Extended or CCBot — collects content to train a model’s underlying knowledge, not to cite you in real time. Blocking a training crawler is a content-rights decision that keeps your pages out of training data; it does not remove you from AI search answers. Blocking a search crawler, by contrast, directly removes you from that engine’s cited sources.
Why AI bots may be blocked — and how to fix it
If a crawler you want is blocked, work through the likely causes in order. Most silent blocks are one of these.
- robots.txt Disallow. A rule like
Disallow: /under the bot’s User-agent group. Fix: add anAllow: /or remove the Disallow for that agent. - Blanket
User-agent: *Disallow. A site-wide block catches every AI bot without its own rule. Fix: add explicit Allow groups for the AI bots you want. - HTTP 401 / 403. Not a robots issue — a WAF, CDN or auth layer is rejecting the bot. Fix: allow the user-agent or its verified IP ranges in your firewall.
- Cloudflare / WAF “block AI bots”. A managed rule blocks AI crawlers regardless of robots.txt. Fix: exempt the AI search bots you want in the WAF, not just in robots.txt.
- Rate limiting. Aggressive limits return 429s and the bot backs off. Fix: raise limits for verified crawler IPs.
- Server errors (5xx). A bot cannot index a page that keeps failing. Fix: resolve the errors and confirm a stable 200.
- X-Robots-Tag / meta robots noindex. The page is crawlable but marked not-indexable. Fix: remove
noindexwhere you want the page eligible. - Redirect loops or unreachable robots.txt. If /robots.txt 5xxs or loops, some bots treat the site as disallowed. Fix: return a clean 200 or 404 for /robots.txt.
How to allow or block AI crawlers in robots.txt
Four starting points, each stating who it is for. Give every bot its own User-agent group: rules do not merge across groups, so a bot named in its own group stops reading the wildcard one entirely — which is how a single Allow: / for OAI-SearchBot silently cancels the site-wide block someone added above it. robots.txt is advisory either way. Compliant crawlers honour it, it is not firewall-level enforcement, and it removes nothing from an index.
For a site that wants maximum reach and has no content-licensing concern.
User-agent: *
Allow: /For a publisher opting out of model training while staying quotable in AI answers.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /When the objection is to OpenAI training specifically, not to AI search.
User-agent: GPTBot
Disallow: /When a site-wide Disallow already exists and ChatGPT needs an exemption from it.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /robots.txt vs llms.txt
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Tell crawlers which paths they may request | Point AI tools to your key content in Markdown |
| Enforceability | Advisory — honoured by compliant bots, not a firewall | Advisory guidance — not an access-control mechanism |
| Who reads it | Search and AI crawlers that follow the standard | AI tools that choose to support it (not universal) |
| Main use | Control crawl access per user-agent | Help AI understand and surface your best pages |
| Limitations | Does not prevent indexing; ignored by non-compliant bots | Emerging convention; no ranking or access guarantee |
Neither file blocks a non-compliant bot, and neither removes a URL from an index. For indexing control, use noindex. Validate your own file with the llms.txt Validator.
Does crawler access guarantee AI visibility?
No, and the gap is bigger than it looks. robots.txt is the only layer this tool can read, and it is the layer least likely to be the real blocker. Above it sit four failures it cannot see: a WAF or bot-management rule that answers the crawler with 403 while your browser gets 200; a page whose content arrives only after JavaScript runs, so the fetch succeeds and returns nothing usable; a noindex that keeps a crawlable page out of the index; and rate limits that turn a crawl into a string of 429s. Clear all five and you are eligible to be retrieved, which is still not the same as being chosen. Retrieval ranks passages against a question, so the page also has to answer one.
AI crawler checker — FAQ
How do I check whether GPTBot can access my website?
Paste the domain above. The checker fetches /robots.txt, then resolves the rule that actually applies to GPTBot: its own User-agent group if the file has one, otherwise the wildcard group — never both, because a bot named in its own group stops reading the wildcard entirely. You get the verdict and the exact line that produced it. GPTBot is the training crawler, so a block here is a content-licensing decision, not a search decision.
What is the difference between GPTBot and OAI-SearchBot?
Different jobs, separate rules. GPTBot collects pages to train OpenAI models. OAI-SearchBot builds the index ChatGPT Search draws on when it answers with links. Blocking GPTBot keeps your text out of training and changes nothing about whether ChatGPT can cite you; blocking OAI-SearchBot removes you from the source pool. Sites that meant to do the first and wrote one blanket rule did the second by accident.
Can ChatGPT access a site that blocks GPTBot?
Yes, and this is the most useful thing on this page. ChatGPT Search reaches pages through OAI-SearchBot for indexing and ChatGPT-User for a live fetch during a conversation. Allow those two, disallow GPTBot, and you have opted out of training while staying fully quotable.
What is ChatGPT-User?
The user-triggered fetcher. It loads one specific page at the moment someone asks about that URL or the model browses on their behalf — it does not crawl your site in bulk. Block it and ChatGPT cannot open your page live in a conversation, even for someone who pasted the link themselves.
What is the difference between PerplexityBot and Perplexity-User?
The same indexing-versus-live-fetch split OpenAI uses. PerplexityBot builds the answer index; Perplexity-User fetches a page on demand when a user points at it. Blocking the first removes you from Perplexity's indexed sources. Blocking only the second leaves you indexed but unreachable in the moment.
Does Google-Extended control Google Search or AI Overviews?
Neither. Google-Extended is a training toggle for Gemini and Vertex AI. Googlebot crawls, ranks and feeds AI Overviews regardless of what you set for Google-Extended — so disallowing it costs you nothing in search and buys you nothing in AI answers. Treat it as a rights decision, not a visibility lever.
How do I allow ClaudeBot in robots.txt?
Give it a group with User-agent: ClaudeBot and either no matching Disallow or an explicit Allow: /. One caveat that catches people: Anthropic runs three agents — ClaudeBot for general crawling, Claude-SearchBot for the search index, Claude-User for live fetches. A rule for ClaudeBot alone leaves the other two on whatever the wildcard group says.
Can Cloudflare block AI crawlers?
Yes, and independently of robots.txt. Cloudflare and other CDN or WAF layers can challenge or reject bots by user-agent, ASN or managed "block AI bots" rulesets. That is why this tool reports "Allowed by robots.txt" and never "AI can reach your site" — the two are different claims, and the second one this check cannot make.
Why does an AI bot receive a 403 response?
Almost never from robots.txt, which asks politely and returns nothing. A 403 comes from a firewall, CDN bot management, a rate limiter, or a server rule rejecting the user-agent or IP range. If your robots.txt passes here and bots still see 403s, the fix is in your WAF configuration, not in this file.
How do I block AI training bots but allow AI search bots?
Disallow the training agents — GPTBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent — and allow the retrieval ones: OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User, Claude-User. Give every one its own User-agent group. Rules do not merge across groups, so this only works when each agent is named explicitly. There is a copy-paste version above.
Can robots.txt prevent indexing?
No, and the mistake is expensive. robots.txt controls crawling; a disallowed URL can still be indexed as a bare link if other pages point at it, and it will look worse for having no content. To keep a page out of an index you need noindex in a meta robots tag or an X-Robots-Tag header — which the crawler has to be allowed to fetch in order to read.
How often should I recheck AI crawler access?
On every deploy that touches robots.txt, after any CDN or WAF change, and after a platform migration — those are where silent blocks come from. Beyond that, quarterly is enough to catch the user-agents vendors add without announcement; the registry behind this tool is re-read against vendor documentation on the same cadence.
This checks one layer. The audit checks all five.
The free AI Visibility Audit runs 47 checks across crawler access, llms.txt, JSON-LD, metadata, content extractability, brand-as-entity signals and current AI visibility, then emails a 0–100 score with the three fixes that move it most.
Run the free AI Visibility Audit