Tool

AI Crawler Access Checker

Paste a robots.txt or enter a domain to see which AI platforms your file lets cite you and which it lets train on your pages. Each verdict names the rule that produced it.

Rules can differ by path, so a site open at the root may still block a crawler from its docs. A pasted file stays in your browser.

robots.txt
⌘↵ to run
0 lines
access.log
{ }

Findings appear here after you check a file.

Method

How this reaches its verdict

citation versus training
  • ChatGPT. OAI-SearchBot governs citation, GPTBot governs training. OpenAI states a site blocking the first "will not be shown in ChatGPT search answers".
  • Claude. Claude-SearchBot governs search, ClaudeBot governs training. Anthropic states that blocking ClaudeBot leaves the other two unaffected.
  • Perplexity. PerplexityBot surfaces and links sites, and Perplexity states it is not used for foundation-model training.
  • Google. AI Overviews and AI Mode are part of Search and use Googlebot. Google-Extended covers Gemini apps and Vertex AI grounding, and does not affect Search.
  • Apple. Applebot powers search features, and Applebot-Extended is a training opt-out only.
matching rules
  • Only the most specific group applies. RFC 9309 selects one user-agent group per crawler. A group naming a crawler replaces the * group for it rather than adding to it.
  • Same-token groups are merged. The checker treats two User-agent: Googlebot blocks as one group, which real files rely on.
  • The longest matching rule wins, and Allow breaks a tie of equal length.
  • An empty Disallow permits everything, and a rule placed before any User-agent line is ignored.
  • Wildcards are honoured. * matches any sequence and $ anchors the end of a path.
  • Matching is case-insensitive on the user-agent token.
assumptions and limits
  • This reads the file, not the network. A CDN or WAF can block a crawler at the edge whatever robots.txt says, and a crawler can ignore the file. Confirm at the edge before concluding a crawler has access.
  • A training opt-out is not a defect. It is an editorial choice, so it is reported separately and never counted against the headline.
  • No score. The headline is a count of citation crawlers allowed. There is no correct number of permitted crawlers, so weighting one would invent a judgement.
  • An absent robots.txt allows everything. A 404 is a real answer, not an error.
  • Agent lists change. Every entry names the document it came from and the date it was read.
Reference

Citation and training are separate controls

Each platform listed above reads at least two robots.txt tokens: one that decides whether it may quote a page in an answer, and one that decides whether the page may train a model. They are separate lines in robots.txt, and a file can permit the first while refusing the second.

The checker catches the mistake in both directions. A file can block citation while its owner believes it only opted out of training, and a file that looks hostile to AI can still let each assistant above quote the site.

Frequently asked questions

What is the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot governs whether ChatGPT may surface a site in search. GPTBot governs training. Blocking GPTBot alone does not affect citation.

Does blocking ClaudeBot stop Claude citing my site?

No. Anthropic states that blocking ClaudeBot leaves Claude-SearchBot and Claude-User unaffected.

Does blocking Google-Extended remove me from AI Overviews?

No. AI Overviews and AI Mode are part of Search and use Googlebot. Google-Extended covers Gemini apps and Vertex AI grounding.

Why did a named group override my wildcard rules?

RFC 9309 applies only the single most specific matching group. Naming a crawler replaces the * group for that crawler rather than adding to it.

Is my pasted file uploaded?

No. A pasted file is evaluated in the page and never transmitted. Only the check-a-domain path contacts the server, carrying the domain alone.