AI Crawlers and robots.txt: How to Allow or Block GPTBot and More

This guide shows you how to handle AI crawlers robots.txt rules so you can block model training if you want to, while keeping the search bots that decide whether AI tools can cite you.

Should I block GPTBot in robots.txt?

Block GPTBot only if you do not want your content used to train OpenAI’s models, because blocking it does not remove you from ChatGPT search. OpenAI’s overview of its crawlers says disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models, and that GPTBot and OAI-SearchBot are controlled independently.

For most small businesses I work with, the decision comes down to this. If your content is your product (a publisher, a course creator, a paid database), blocking training bots is reasonable. If your content exists to win customers (a service business, a store, a consultant), being known to AI models has some upside, and many owners choose to allow both. There is no single right answer; it is a business decision, not an SEO trick.

Whatever you choose, write it down. I keep a short note in each client’s SEO file listing which AI bots are allowed and why, so a developer or plugin update six months later does not quietly reverse the decision. Robots.txt files get overwritten more often than people expect, especially during redesigns and hosting moves.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used for model training, while OAI-SearchBot is used to surface websites in ChatGPT search results. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

The same training versus search split exists at other companies. The table below lists the main AI user agents from each provider’s own documentation.

Anthropic’s help article on how site owners can block its crawlers describes ClaudeBot, Claude-User and Claude-SearchBot, and notes that blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude’s search results. Google’s documentation explains that Google-Extended limits use for AI training and grounding in Google’s other systems, while AI Overviews and AI Mode in Search are governed by Googlebot and normal snippet controls.

Does blocking AI crawlers hurt my visibility in ChatGPT?

Blocking GPTBot does not hurt your visibility in ChatGPT search, but blocking OAI-SearchBot does, because OpenAI says opted out sites are not shown in ChatGPT search answers. The common mistake I see is a blanket rule or a security plugin that blocks every bot with “AI” or “GPT” in its name, which silently removes the site from AI answers.

Firewalls cause the same problem. A CDN bot setting may return 403 to unknown crawlers even when robots.txt allows them. OpenAI and Perplexity both publish IP ranges for their bots so you can allowlist them properly. Check both layers: robots.txt rules and what your server actually returns.

Remember that robots.txt is public and only controls crawling by bots that choose to respect it. It is not a security tool. If you have content that must stay private, put it behind a login rather than relying on a Disallow line, because a Disallow rule also tells everyone exactly where that content lives.

What robots.txt rules should I use?

Use one group per user agent, list the bots you want to block with Disallow: /, and leave search bots allowed. This is the template I start from for a business that wants AI citations but no training use:

• User-agent: GPTBot then Disallow: /
• User-agent: ClaudeBot then Disallow: /
• User-agent: Google-Extended then Disallow: /
• User-agent: OAI-SearchBot then Allow: /
• User-agent: Claude-SearchBot then Allow: /
• User-agent: PerplexityBot then Allow: /
• User-agent: * then your normal rules, plus your Sitemap: line

Then test it. My free robots.txt tester lets you check a URL against a specific user agent, and my guide to the blocked by robots.txt status explains what happens when Googlebot itself is caught by a rule.

1. Back up your current robots.txt.
2. Add the groups above, adjusted to your own choice for each bot.
3. Test your homepage and one key page against each user agent.
4. Check your CDN or firewall bot settings for blocks on the bots you allowed.
5. Wait about 24 hours, which is the time OpenAI and Perplexity say their systems may need to reflect changes.
6. Review server logs a week later to confirm the allowed bots return 200 responses.

How do I check which AI bots visit my site?

The most reliable way to see which AI bots visit your site is to read your server access logs and filter by user agent. Analytics tools such as GA4 do not record most crawlers, because bots usually do not run JavaScript, so a site can be crawled daily by OAI-SearchBot and show nothing in analytics.

Most hosts let you download raw access logs from the control panel. Search the file for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and ChatGPT-User, and note three things for each: how often it visits, which URLs it requests, and which status codes it receives. A bot you allowed that keeps getting 403 or 503 responses is blocked somewhere outside robots.txt.

Be aware that anyone can fake a user agent string. If a “GPTBot” is hammering your site from an unusual network, compare its IP address with the ranges the provider publishes before you trust it. OpenAI and Perplexity both publish JSON lists of their IP ranges for this purpose.

On WordPress, also check your security plugin’s blocked requests log. In my audits, plugins that block “bad bots” by default are the most common reason a site disappears from AI search tools without the owner ever touching robots.txt.

I repeat this log check once a month for sites where AI visibility matters. It takes about 15 minutes, and it is the only way to be sure that the rules you wrote are the rules your server actually enforces.

What does a fixed setup look like in practice?

If you want this checked as part of a wider audit, it is included in my technical SEO service.

Frequently asked questions