llms.txt Generator
An llms.txt generator is a tool that writes the /llms.txt file proposed on llmstxt.org. The logic is simple:…
AI bot user agent checker
Paste user agents or log lines to see which AI crawlers visited and what each one does.
Paste raw access log lines or plain user agent strings. For log lines the tool reads the last quoted value, which is the user agent in the common combined log format.
Result
0Lines checked
0AI bot hits
0Other or human
| Bot | Owner and purpose | robots.txt |
|---|
The block rules are a starting point only. Remove any bot you want to keep, for example search bots that can cite and link to you.
The free gptbot user agent runs in your browser, needs no sign-up and never sends what you type anywhere.
Quick answer
The GPTBot user agent contains the token GPTBot, for example "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot". GPTBot collects training data, while OAI-SearchBot powers ChatGPT search and ChatGPT-User fetches pages for users. Block or allow each one by its own robots.txt token, and verify IPs.
What is a ai bot user agent checker?
An AI bot user agent checker is a tool that scans user agent strings for the documented tokens of AI crawlers and fetchers. It matches each line against names published by OpenAI, Anthropic, Perplexity, Meta, Google, Apple and Common Crawl, then shows the owner, purpose and robots.txt behaviour.
How do you use the AI Bot User Agent Checker?
- Export a day of raw access logs from your host, CDN or server, or copy single user agent strings.
- Paste the lines into the box; each line can be a full log entry or just the user agent.
- Read which AI bots were found, how many hits each made and what the owner says the bot is for.
- Decide per bot: training crawlers, search crawlers and user triggered fetchers have different effects.
- Copy the robots.txt block rules, delete the bots you want to keep, and add the rest to your robots.txt.
What is the GPTBot user agent string?
OpenAI lists the GPTBot user agent as "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot", and the version number can change. Match on the GPTBot token, not the whole string. OpenAI's crawler overview also lists OAI-SearchBot and ChatGPT-User with their own tokens.
Does blocking GPTBot remove my site from ChatGPT search?
No. Blocking GPTBot only signals that your content should not be used to train OpenAI models. ChatGPT search uses OAI-SearchBot, which you control with its own robots.txt group. Sites that block OAI-SearchBot will not appear in ChatGPT search answers, though OpenAI says they may still show as navigational links.
Which AI bots ignore robots.txt?
User triggered fetchers can ignore robots.txt because a person asked for the page. OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says Meta-ExternalFetcher may bypass it. Training and search crawlers such as GPTBot, ClaudeBot and PerplexityBot are documented as following robots.txt.
Quick reference
| robots.txt token | Owner | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content that may train foundation models |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search |
| ClaudeBot / Claude-SearchBot | Anthropic | Training data / search quality |
| PerplexityBot | Perplexity | Indexes and links sites in Perplexity answers |
| Meta-ExternalAgent | Meta | Trains AI models or indexes content |
| Google-Extended | Control token for Gemini training and grounding; no separate crawler |
Worked example
Example
Example: a log sample of 6 lines contains GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Meta-ExternalAgent and one Chrome visitor, so the checker shows 5 AI bot hits and 1 other. For a services site that wants AI citations, I would keep PerplexityBot, OAI-SearchBot and Claude-SearchBot allowed and only consider blocking the training crawlers GPTBot, ClaudeBot and Meta-ExternalAgent.
Learn more
Frequently asked questions
Is GPTBot the same as ChatGPT-User?
No. GPTBot crawls on its own to gather content that may train OpenAI models. ChatGPT-User only visits a page when a ChatGPT user or a custom GPT asks for it, does not crawl the web automatically and is not used to decide search inclusion.
Why can't I find Google-Extended in my logs?
Google-Extended is a robots.txt control token, not a crawler. Google crawls with its normal user agents and uses the token only to decide whether content may be used for Gemini training and grounding. Google says it does not affect inclusion or ranking in Google Search.
Can someone fake the GPTBot user agent?
Yes. Any script can send a GPTBot or ClaudeBot string. OpenAI, Anthropic, Perplexity and Google publish IP lists or verification methods, so compare the request IP with those lists before you trust a hit or build firewall rules around it.
Should I block AI crawlers?
It depends on your goal. Blocking training crawlers limits model training use of your content. Blocking AI search crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot can stop those tools from citing and linking to you, which most business sites want to keep.
What is Applebot-Extended?
Applebot-Extended is a robots.txt token Apple uses to let publishers opt out of their content training Apple's foundation models. Apple says it does not crawl webpages; pages are still crawled by Applebot, which supports Siri, Spotlight and Safari search.
Does this checker upload my log files?
No. The checker runs entirely in your browser and reads only the text you paste. Nothing is uploaded or stored, so you can check raw access logs that contain IP addresses without sending them to any server.
Sources
Facts checked against these official pages. Read how I check facts in my editorial policy.
Related free tools
All free toolsAI Crawler robots.txt Generator
An AI crawler robots.txt generator is a tool that turns your allow or block choice for each AI user agent…
User Agent Parser
A user agent parser is a tool that turns a raw User-Agent string into readable parts. It matches known tokens…
Want a full audit instead?
Send your website and I will check it for you, free.