How to verify Googlebot (and every other crawler) from your own logs

You can verify Googlebot with two DNS lookups and nothing else: reverse-resolve the IP address from your log, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then forward-resolve that hostname and check it returns the same IP. The user agent string in the request is not evidence. Anyone can send one.

Before you start

You need a log that records the client IP address: a server access log, a CDN log, or an edge function log. Search Console will not do — its crawl stats only show requests Google already knows are its own, so it can never show you the traffic you are suspicious about. If your host does not expose logs at all, the checks below are unavailable to you, and the honest next step is to find out whether your CDN can turn them on before spending time on anything else.

Why the user agent header proves nothing

A user agent is a text field the client fills in. Copying Googlebot's exact string takes one line of code, and scrapers do it constantly to get past rules that filter by name. Every allow-list built on the name alone is therefore a rule that helps the people it was meant to stop, because well-behaved crawlers identify themselves honestly and badly-behaved ones borrow whichever name gets through.

What cannot be borrowed is the connection itself. To receive your response, the client has to complete a TCP handshake from an address it actually controls, and that address is either inside the crawler operator's own network or it is not. Both verification routes below are ways of asking that one question.

A user agent is a claim about who is asking. The IP address is the part of the request that has to be true for the answer to arrive.

How to verify Googlebot in two DNS lookups

Google documents a two-step check, and it is the same procedure whether the request claims to be Googlebot, AdsBot or Google Site Verifier. Run a reverse lookup on the address, read the hostname, then run a forward lookup on that hostname and compare. If either step disagrees, the request was not from Google.

1. Run host <ip> on the address from your log and read the hostname it returns.
2. Check the hostname's domain. Google's documentation, read on 2026-08-23, says it must be googlebot.com, google.com or googleusercontent.com.
3. Run host <hostname> on what step 1 returned and confirm the address matches the one you started with.

# Google's own worked example, from the verification documentation
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Step 3 is the one people skip, and skipping it removes the whole point of the exercise. Reverse DNS records are set by whoever controls the address block, so an attacker can point their own reverse record at a convincing hostname. What they cannot do is make Google's forward DNS resolve that hostname back to their address. The forward lookup is the half that is out of their hands.

The hostname shape also tells you which part of Google you are talking to. Common crawlers such as Googlebot resolve to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com. Special-case fetchers resolve to rate-limited-proxy-*.google.com. User-triggered fetchers resolve to *.gae.googleusercontent.com or google-proxy-*.google.com. Those three groups obey robots.txt differently, so the hostname is worth reading rather than just pattern-matching.

The deliverable: every published crawler IP list in one table

For crawlers that do not document a reverse DNS pattern, the route is to match the address against a list the operator publishes. Five operators publish machine-readable files. Every row below was fetched and parsed on 2026-08-23; the timestamps are the creationTime field inside each file, not the date we downloaded it.

All nine files share the same shape — a creationTime string and a prefixes array of CIDR blocks — so one script reads all of them. Save this, point it at an address from your log, and it answers in one line.

#!/usr/bin/env bash
# usage: ./whois-bot.sh 66.249.66.1
IP="$1"
for URL in \
  https://developers.google.com/static/crawling/ipranges/common-crawlers.json \
  https://openai.com/gptbot.json \
  https://openai.com/searchbot.json \
  https://openai.com/chatgpt-user.json \
  https://claude.com/crawling/bots.json \
  https://www.bing.com/toolbox/bingbot.json \
  https://www.perplexity.ai/perplexitybot.json
do
  curl -s "$URL" | python3 -c '
import ipaddress, json, sys
ip = ipaddress.ip_address(sys.argv[1])
for p in json.load(sys.stdin)["prefixes"]:
    net = p.get("ipv4Prefix") or p.get("ipv6Prefix")
    if net and ip in ipaddress.ip_network(net):
        print("MATCH", sys.argv[2], net)
' "$IP" "$URL"
done

Two things in that table are worth acting on rather than reading past. Bingbot's file carries a 2024-01-03 timestamp, more than two years older than any other row, so an address that fails against it has not necessarily failed. And Anthropic's own help page, dated 2026-04-07, says that blocking by IP address "may not work correctly or persistently guarantee an opt-out" — the list is published for identification, and the supported way to refuse the crawler is robots.txt. Use these files to answer "who was that", not as a firewall policy.

Where this goes wrong

Three failure modes account for almost every broken version of this check, and all three are shortcuts rather than mistakes in the DNS work itself.

• Verifying the name instead of the address. If your rule reads the user agent and stops there, it is a rule that filters the honest crawlers only.
• Stopping after the reverse lookup. The reverse record is controlled by the address owner; without the forward lookup you have verified nothing.
• Blocking every address that fails a match. A miss can mean the operator added capacity this morning, or that the file has not been regenerated since 2024. Log it, look at what it did, then decide.

Before you build any rule on top of this, be clear about which crawlers you meant to let in at all — that decision is the subject of which AI crawlers to allow, and it comes first. If you only want to know whether the well-behaved ones can reach your pages right now, the fastest route is the AI crawler accessibility check.

What this check cannot tell you

It identifies the sender. It does not tell you what the sender did with the page, and no verification step can. One company also runs several crawlers with different jobs and different consequences for blocking them, which is why an address that verifies is only half an answer — see how GPTBot and OAI-SearchBot differ before you write a rule per company.

There is also a gap we cannot close from a log file. Only Google documents a reverse DNS naming pattern; for the other operators the published IP file is the only route on offer, and if an operator adds addresses faster than it regenerates that file, a genuine request will fail the check. We do not know how often that happens, because the operators do not publish regeneration schedules and a miss looks the same either way.

Common questions

How do I verify Googlebot without server logs?

You cannot. Every method needs the client IP address, and that only exists in a log or an edge rule. Search Console's crawl stats report is not a substitute — it shows Google's own verified traffic, so the requests you doubt are exactly the ones missing from it.

Is checking the IP range as good as the DNS check?

For Google, the DNS check is better, because it is generated live rather than from a file that has to be regenerated. For OpenAI, Anthropic, Bing and Perplexity the IP file is what exists, so it is what you use.

What should I do when a request fails verification?

Treat it as unidentified traffic rather than as an attack. Look at what it requested and how fast. A single fake Googlebot reading three pages is noise; a thousand requests a minute is a rate-limiting problem, and rate limiting is the correct response to it — not a name-based block, which the same client will simply route around.

Do AI crawlers have reverse DNS records like Googlebot?

Their documentation does not promise one. OpenAI's crawler page points to published IP files as the way to allow its traffic, and Anthropic's help page points to its own list. We tested this only against what those pages say, not by resolving live addresses from each operator.