Log file analysis for SEO: how to see which AI crawlers actually visited

Log file analysis for AI crawlers means reading the User-Agent and IP fields your web server already recorded, then separating requests that came from a real AI crawler from requests that only claimed to be one. Your access log answers a question a reachability check cannot. Not "could GPTBot reach this page", but "did it, on which day, and how many times". A 200 in a test proves the door opens. The log shows who walked through it.

Read this first

You need read access to a server access log — nginx, Apache, Caddy, or a managed host that exposes one — and a terminal. That is the whole prerequisite. If your host does not give you logs, this chapter has no substitute: no third-party tool can see your raw requests, because they never reached your server.

Two chapters sit next to this one. If you have not yet checked whether AI crawlers can reach you at all, start with how to check if AI can read your site, which tests the door from the outside. And if you want the measured version of the question "who actually shows up", that is a separate post: what AI crawlers actually see on your site covers the 85-request check we ran across five sites.

What log file analysis can and cannot tell you

It can tell you which user agents requested which paths, at what times, with what status codes, and from which IP addresses. It cannot tell you whether the content was later used in an answer or in training. And it cannot confirm identity on its own: the User-Agent string is self-reported, and anyone can copy it. Only a matching IP turns "a request that says GPTBot" into "GPTBot".

That distinction is the whole reason this is a chapter and not a one-line tip. A log line has two halves. The half people read is the User-Agent. The half that decides whether the line is true is the IP address.

Why a log beats a user-agent check

A reachability check and a log measure different things, and the gap between them is where most wrong conclusions live. A check sends a request with a spoofed User-Agent and reports what your stack did. A log records what real clients did, unprompted.

The two can disagree in both directions. You can pass a check and still get no AI crawler traffic, because reachable is not the same as visited. You can also fail a check and still find crawler traffic, because the path a test takes is not always the path a crawler takes — a CDN edge rule can treat a HEAD request from a data centre differently from a GET from a real crawler.

Logs also carry the one field a check can never produce: time. A check gives you a yes or no at one moment. A log gives you a count per day per bot, which is the only way to tell a crawler that visits weekly from one that came once and left.

Do it in this order

Five steps. Each one has something you can check before moving on.

1. Find the log and confirm its format. For nginx the default is the combined format; for Apache it is the NCSA extended/combined format. Apache documents both as %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i", where the last quoted field is the User-Agent and the second-to-last is the referrer. Done when you can point at one line and name every field.
2. Count requests by User-Agent before you filter anything. Splitting a combined line on the quote character puts the User-Agent in the sixth field. Done when you have a top-30 list of every agent that hit the site, not just the ones you expected.
3. Pull the AI crawler tokens out of that list. The exact strings are in the table below. Done when each token has a request count and a first-seen date.
4. Verify the IPs you plan to believe. Reverse DNS for Google, the published JSON range files for OpenAI. Done when every line you are about to quote has either passed a forward-confirmed reverse lookup or is labelled unverified.
5. Record the window, not just the total. A count means nothing without the number of days it covers. Done when your note reads like "412 requests, 2026-08-01 to 2026-08-31, 5 sites" rather than "lots".

The deliverable: one command and two tables

This one-liner turns a combined-format log into a count of every agent that visited. It assumes no escaped quotes inside the User-Agent field, which is true for the crawlers below and false for some scrapers.

awk -F'"' '{print $6}' access.log \
  | sed 's/;.*compatible;//' \
  | sort | uniq -c | sort -rn | head -30

The sed step is there because browser agents bury the real product token behind a compatibility prefix. It is a blunt cut; keep the raw sixth field too, so you can go back to it when a count looks wrong.

The second deliverable is a field table. Most people know the User-Agent field and stop there, which is why their first pass misses the paths.

The third deliverable is the token list. These are the strings the operators publish, read on 22 September 2026 from their own documentation. It is not a complete list of every AI crawler, and new ones appear without notice — which is exactly why step 2 counts everything first.

Two of those rows are load-bearing. Google-Extended is a control token, not a crawler: Google's documentation states it "doesn't have a separate HTTP request user agent string", so it will never appear in your log and its absence is not a failure. And ChatGPT-User is triggered by a person, so its requests are not an automatic crawl — the same documentation says it "is not used for crawling the web in an automatic fashion".

What goes wrong

You trust the User-Agent. This is the default mistake and it produces a confident, wrong answer. Anyone can send GPTBot/1.4 in a request header, and scrapers do. Google's own guidance is explicit that verification means two DNS lookups: a reverse lookup on the IP, then a forward lookup on the name it returns, with the two having to match. For OpenAI you match the IP against the published range file instead. For a crawler that publishes neither, you cannot verify it, and the honest move is to write "unverified" next to the count rather than to drop it or to believe it.

You count robots.txt fetches as page crawls. OpenAI's documentation notes that when it fetches robots.txt it "may add a robots.txt marker to the user-agent string to help site owners distinguish those requests from requests for other resources, especially when logs do not include paths". If your log format drops the path, a robots.txt fetch and a page fetch look identical. Split by path before you report a crawl count.

You read a window that is too short. A seven-day log cannot answer a thirty-day question, and log rotation silently truncates the window without warning. State the first and last timestamp you actually counted, not the period you intended to count.

Common questions

Can I trust the user agent in my logs?

No. Treat it as a claim until the IP matches a published range or passes forward-confirmed reverse DNS. Logs are records, not proof.

How do I check whether GPTBot visited my site?

Grep your log for the token GPTBot, then match every matching IP against the range file at openai.com/gptbot.json. A token match without an IP match is a different request that copied the name.

Is Googlebot verification different from the others?

Yes, and it is the stricter case. Google does not ask you to match a static list first; it asks for a reverse DNS lookup, a check that the name ends in googlebot.com, google.com or googleusercontent.com, and then a forward lookup that returns the original IP. It also publishes JSON range files if you prefer to match addresses directly.

What is the difference between GPTBot and OAI-SearchBot?

They are separate tokens with separate jobs. OpenAI's documentation describes OAI-SearchBot as the one "used to surface websites in search results in ChatGPT's search features", and GPTBot as the one that crawls content "that may be used in training". Allowing one and blocking the other is a supported configuration, not a contradiction.

How far back do I need logs to answer "did AI crawlers visit"?

Long enough to cover a full crawl cycle, which depends on your site and we cannot give you a number that holds for all of them. The safe default is thirty days. If you only have seven, say so when you report the count — a shorter window undercounts, and it never overcounts.

The boundary worth stating plainly: a log proves a fetch happened. It does not prove the fetched content was used, ranked, or quoted, and no amount of counting fixes that. What it does is replace a guess about whether AI crawlers reach you with a dated record — and that record is the input to every decision that follows. When you are ready to turn the record into a change and get it published, that is the part QueryWin works on.