Robots.txt Generator
A robots.txt generator is a tool that writes the plain text file crawlers read at the root of your site…
Free AI crawler robots.txt generator
Choose which AI bots may crawl your site and get clean robots.txt rules in seconds.
Result
0Bots blocked
0Bots allowed
0Lines
Merge these rules into your existing robots.txt at the site root. robots.txt is a request that well behaved bots follow, not access control.
The free ai crawler robots.txt generator runs in your browser, needs no sign-up and never sends what you type anywhere.
Quick answer
An AI crawler robots.txt generator writes User-agent and Disallow rules for AI bots such as GPTBot, ClaudeBot and PerplexityBot. A common setup blocks training crawlers but allows search bots like OAI-SearchBot so you can still be cited. Blocking Google-Extended does not remove you from Google Search or AI Overviews.
What is a ai crawler robots.txt generator?
An AI crawler robots.txt generator is a tool that turns your allow or block choice for each AI user agent into robots.txt groups. Each blocked bot gets a User-agent line and Disallow: /, while allowed bots follow your normal rules or the private paths you list.
How do you use the AI Crawler robots.txt Generator?
- Pick a preset: allow all, block training but allow AI search, or block all AI bots.
- Adjust each bot to Allow or Block; the note beside each name says what it does.
- List any private paths that allowed bots should still skip, such as /wp-admin/.
- Add your sitemap URL and choose whether to include a general User-agent: * group.
- Copy or download the rules, merge them into your live robots.txt, and test a URL with a robots.txt tester.
How do I block AI crawlers with robots.txt?
Add a group for each bot with its user agent token and Disallow: /, for example User-agent: GPTBot then Disallow: /. Each vendor lists its tokens, such as OpenAI's crawler overview. robots.txt is a request, not a lock, so well behaved bots follow it.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended is a robots.txt control token, not a crawler, and Google's list of common crawlers says it does not affect inclusion or ranking in Google Search. It controls use for Gemini training and grounding. AI Overviews use normal Googlebot and follow snippet controls such as nosnippet.
Should I block GPTBot and ClaudeBot?
Block training bots such as GPTBot and ClaudeBot only if you do not want your content used for model training; it does not stop ChatGPT or Claude search citations, which use OAI-SearchBot and Claude-SearchBot. Most businesses that want AI visibility allow the search bots at least.
Quick reference
| User agent token | Company | Purpose |
|---|---|---|
| GPTBot / OAI-SearchBot / ChatGPT-User | OpenAI | Training / ChatGPT search / user fetches |
| ClaudeBot / Claude-SearchBot / Claude-User | Anthropic | Training / search quality / user fetches |
| PerplexityBot / Perplexity-User | Perplexity | Search index / user fetches |
| Google-Extended | Token for Gemini training and grounding | |
| Applebot-Extended | Apple | Token for Apple model training |
| CCBot / meta-externalagent | Common Crawl / Meta | Open web archive / AI training and indexing |
Worked example
Example
Example: a recipe blog wants citations in ChatGPT, Claude and Perplexity answers but no training use. It blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent with Disallow: /, and allows OAI-SearchBot, Claude-SearchBot and PerplexityBot except /members/. That is 6 blocked groups and 3 allowed groups in one file.
Learn more
Frequently asked questions
Do AI crawlers respect robots.txt?
The major vendors say their crawlers follow robots.txt, but user triggered fetchers may not. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. To truly block access, use server or firewall rules as well.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's foundation models, while OAI-SearchBot finds pages to show in ChatGPT search results. You can block GPTBot and still allow OAI-SearchBot, so your pages can appear and be linked in ChatGPT answers.
Will blocking AI bots hurt my Google rankings?
No. Rules for GPTBot, ClaudeBot, PerplexityBot and similar bots do not affect Googlebot, and Google says the Google-Extended token is not a Search ranking signal. Just make sure no rule accidentally blocks Googlebot or the User-agent: * group.
How long until AI crawlers see my new robots.txt?
Most crawlers cache robots.txt for a while before fetching it again; Meta says changes can take up to 24 hours to apply. Content already collected before the change is not removed automatically, so robots.txt only affects future crawling.
Why is Bytespider not in this generator?
This tool only lists bots whose names and purposes are documented on the vendor's own pages. If you want to block another user agent, such as Bytespider, add it as a custom line using the exact token you see in your server logs.
Do I need llms.txt if I allow AI crawlers?
No. llms.txt is an optional proposal that suggests pages for AI tools to read, while robots.txt controls access. Google says it does not use llms.txt. You can add one as a cheap extra, but allowed crawlers can read your site without it.
Sources
Facts checked against these official pages. Read how I check facts in my editorial policy.
Related free tools
All free toolsllms.txt Generator
An llms.txt generator is a tool that writes the /llms.txt file proposed on llmstxt.org. The logic is simple:…
AI Bot User Agent Checker
An AI bot user agent checker is a tool that scans user agent strings for the documented tokens of AI crawlers…
Coming 30 OctWant a full audit instead?
Send your website and I will check it for you, free.