How to Find AI Bots in Your Server Logs (and Verify Them)

This guide is for site owners, developers and SEOs who want to know whether ChatGPT, Claude, Perplexity and other AI systems actually visit their pages. You will learn which user agents to search for, how to tell training bots from search bots, how to confirm a visit is genuine and how to read AI bots in server logs next to your robots.txt rules.

Why look for AI bots in your server logs?

Server logs are the only record of every request an AI bot makes to your site. Search Console covers Google only, and analytics tags record visitors that run their JavaScript, which bots often do not. Your access log, by contrast, writes one line per request with the IP address, time, URL, status code and user agent.

That record answers practical questions. Do AI search bots reach the pages you want cited? Does your CDN or firewall return 403 errors to them? Did a training bot keep crawling after you blocked it? You cannot answer any of these from robots.txt alone, because robots.txt only says what you asked for, not what happened.

Which AI bot user agents should you search for?

Search for the product token each company documents, not the full user agent string, because version numbers change. The main ones belong to OpenAI, Anthropic and Perplexity, and each company runs separate bots for training, search and user requests. OpenAI, for example, says OAI-SearchBot “is for search”, while GPTBot collects content that “may be used in training” its models.

Two names you will never find are Google-Extended and Applebot-Extended. Google says “Google-Extended doesn’t have a separate HTTP request user agent string”, and Apple says “Applebot-Extended does not crawl webpages”. Both are robots.txt controls only. For Google Search, including its AI features, Google’s AI features documentation says your Googlebot rules are the control, so that crawling shows up as ordinary Googlebot hits.

How do you find AI bot hits in an access log?

Download the raw access log from your host, then filter the lines by user agent. On most hosts the logs sit in the control panel under logs or statistics, and many keep only a few days, so save copies regularly. If your site uses a CDN, the CDN may answer some requests before they reach your server, so check its logs too.

The quickest route is to paste the lines into the server log analyzer. It reads the Apache and Nginx combined format, labels each bot by purpose (search, training, user fetch or SEO tool) and counts 2xx, 3xx, 4xx and 5xx responses per bot. Everything runs in your browser, which matters because log lines contain visitor IP addresses. On a server, a command such as grep -i "gptbot" access.log does the same job for one bot at a time.

If you only have a user agent string from a report or a support ticket, the AI bot user agent checker tells you which company and bot it claims to be.

How do you verify that an AI bot visit is real?

Check the IP address, because any client can send any user agent string. The method depends on the company:

• Google. Google’s verifying Googlebot guide says to run a reverse DNS lookup with the host command, check that the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP. Google also publishes its IP ranges as JSON files, such as common-crawlers.json.
• OpenAI. OpenAI publishes separate IP lists for each bot: gptbot.json, searchbot.json and chatgpt-user.json on openai.com.
• Anthropic. Anthropic’s crawler help page links to a bots.json list and says a request from an IP on that list comes from Anthropic.
• Perplexity and Apple. Perplexity publishes perplexitybot.json and perplexity-user.json. Apple says Applebot traffic is identified with reverse DNS in the applebot.apple.com domain or with its published CIDR list.

A hit that claims to be GPTBot from an IP outside OpenAI’s list is not OpenAI. Treat it as an unknown scraper when you read the numbers.

How do you compare log hits with your robots.txt?

Test each bot you found against your robots.txt, then compare the result with what the log shows. Paste your file into the AI crawler robots.txt checker and test the paths the bots requested. Three patterns are worth acting on:

• Blocked but still crawling. A training bot fetching normal pages after you blocked it. First check timing: OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust, and Perplexity says changes can take up to 24 hours. Fetching /robots.txt itself is normal, because that is how bots read your rules.
• Allowed but getting errors. A search bot you allow receives 403 or 5xx responses. That usually points to a firewall, bot protection or CDN rule, not robots.txt.
• User fetches you cannot block. OpenAI says that for ChatGPT-User “robots.txt rules may not apply”, and Perplexity says Perplexity-User “generally ignores robots.txt rules”. These visits follow a person’s request.

For the reasons to allow or block each bot, read my guide to AI crawlers and robots.txt. To see whether those visits turn into real visitors, track ChatGPT traffic in GA4, since log hits show crawling, not clicks.

Worked example

Frequently asked questions

This guide is part of my technical SEO audit checklist.

Want a crawl and log review of your site? See my technical SEO service.