AI Crawler robots.txt Generator
An AI crawler robots.txt generator is a tool that turns your allow or block choice for each AI user agent…
Paste the file from yoursite.com/robots.txt. Matching follows Google's documented rules. Nothing is fetched or sent.
Result
0Bots checked
0Allowed
0Blocked
| Bot and purpose | Group | Deciding rule | Result |
|---|
The free AI crawler robots.txt checker reads your robots.txt and tells you, for 14 search engine and AI bots, whether each one may fetch a given URL and which rule decides it. It uses the matching rules Google documents and runs in your browser.
Quick answer
An AI crawler robots.txt checker tests your robots.txt against bots like GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. Each bot follows the group that names it, or the * group if none does, and the longest matching rule wins, with allow winning a tie. Blocking a training bot does not block search bots.
What does an AI crawler robots.txt checker do?
An AI crawler robots.txt checker is a tool that applies your robots.txt rules to a list of AI and search bots and shows which ones may crawl a URL. AI companies now run several bots with different jobs. OpenAI, for example, separates GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for user requests. Blocking one does not block the others, so a quick check shows whether your file does what you meant.
The tool follows Google's robots.txt rules: a bot uses "the group with the most specific user agent that matches", specific groups and the * group "are not combined", the longest matching path wins and, for conflicts, "Google uses the least restrictive rule". Other bots should follow the same standard, RFC 9309.
How do you use the AI Crawler Robots.txt Checker?
- Open yoursite.com/robots.txt in a browser, copy everything and paste it into the box.
- Type the URL path you want to test, such as / for the home page or /blog/ for your blog.
- Read the table. Each bot shows which group it used, the rule that decided the result and Allowed or Blocked.
- Check the notes for bots that may ignore robots.txt and for tokens that do not crawl.
- Change the file in the box until the result matches your plan, then update the real robots.txt on your server.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses GPTBot for content that may train its models and OAI-SearchBot to "surface websites in search results in ChatGPT's search features". If you block GPTBot but allow OAI-SearchBot, your pages can still appear and be cited in ChatGPT search. Anthropic also has separate bots for training (ClaudeBot), search (Claude-SearchBot) and user requests (Claude-User). Perplexity's bots are for search and user requests only, not training.
Does Google-Extended control AI Overviews?
No. AI Overviews are part of Google Search, which your Googlebot rules control. Google-Extended controls whether content Google crawls may be used for Gemini models and grounding. Google's crawler documentation says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It is a token only and has no user agent string of its own, so it never appears in your logs. Your Googlebot rules decide what Google Search can crawl.
Which bots may ignore robots.txt?
Bots that act on a direct user request. OpenAI says that for ChatGPT-User, "because these actions are initiated by a user, robots.txt rules may not apply", and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Robots.txt is a request, not access control. To keep content private, use a login or block it on the server, not robots.txt.
Quick reference: rule matching
| robots.txt situation | What the bot does |
|---|---|
| A group names the bot | Uses only that group, ignores the * group |
| No group names the bot | Uses the * group |
| Several groups name the same bot | Combines their rules |
| Two rules match the path | Longest path wins |
| Allow and Disallow of equal length | Allow wins (least restrictive) |
| Disallow: with an empty value | Ignored, allows everything |
| * in a path / $ at the end | Any characters / end of the URL |
Worked example
Example
Example: the default robots.txt has 4 groups. Tested at /, 11 of 14 bots are allowed and 3 are blocked: GPTBot and CCBot (they share one group with Disallow: /) and Google-Extended. ClaudeBot is allowed at / because its own group only blocks /private/, and the * group no longer applies to it. Test /wp-admin/admin-ajax.php and Googlebot is allowed: the 24 character Allow rule beats the 10 character Disallow: /wp-admin/.
Learn more
Frequently asked questions
Why does ClaudeBot ignore my * rules?
Because a group names it. A bot that matches a specific group uses only that group, and Google says specific groups and the * group are not combined. Repeat any * rules you still want inside the bot's own group.
Does the tool fetch my live robots.txt?
No. Browsers cannot read another site's files from this page, so you paste the file yourself. That also lets you test a changed version before you upload it, which is the safest way to edit robots.txt.
Is robots.txt case sensitive?
User agent names are not case sensitive, so gptbot and GPTBot match the same bot. Paths are case sensitive, so Disallow: /Private/ does not block /private/. The tool follows both rules.
Should I block AI training bots?
It is a business choice. Blocking training bots such as GPTBot or ClaudeBot keeps your content out of future training data. Keep AI search bots allowed if you want your pages cited in ChatGPT, Claude or Perplexity answers.
Why is a bot I blocked still in my logs?
Bots fetch robots.txt itself to read the rules, so seeing them request that file is normal. Requests from user triggered agents may also continue, and some bots fake their user agent. Check your logs with the server log analyzer.
What is the maximum size of robots.txt?
Google enforces a limit of 500 KiB and ignores content after that. Most robots.txt files are only a few lines long, so this only matters for very large sites with long lists of rules.
Sources
Checked 22 Oct 2026. Facts checked against these official pages. Read how I check facts in my editorial policy.