Fake Googlebot in reverse DNS, and what crawler IP lists miss

A user agent that says Googlebot proves nothing, since any client can send one. Google, Bing and Apple each give two ways to check a request that claims to be their crawler: look up the reverse DNS name of its address and then look that name up forward, which has to return the same address, or match the address against a list of IP ranges the operator publishes (Google, Bing, Apple).

We ran both checks on every IPv4 address whose reverse DNS names one of the three crawlers, with a name ending in googlebot.com, search.msn.com or applebot.apple.com. The names come from IP RDNS, our database of the PTR names our internet-wide scans find, as it stood on October 16, 2026. We ran each name's forward lookup that evening and downloaded the three operators' lists at 22:26 UTC.

Every named address in a list also passed the forward lookup. Where the two checks disagree, it's over confirmed addresses outside the lists: 33,401 that the operator's own DNS names as its crawler and its list turns away. A check that stops at the reverse name, with no forward lookup, lets in 1,387 more whose forward lookup fails.

76 forged Googlebot names

76 of Googlebot's 698 failures are outside Google's networks: on 30 other networks, and two in space no network announced that day. 70 are on networks our ASN database types as hosting. Hetzner's carry 10, and Leaseweb, M247 and IONOS are among the rest. Each still answered with its Googlebot name when we looked it up again at 22:45 UTC.

66 of the names don't exist forward at all, and 60 of those spell the server's own address after crawl-, the way Google's names do. The plainest case is a run of 28 consecutive addresses on one Russian hosting network, AS49063: from 178.20.236.99 to 178.20.236.126, each address's reverse DNS is crawl-178-20-236-<n>.googlebot.com, its own address, and none of the 28 names has a forward record.

The other 10 copy a real Googlebot name, one that does resolve. Four of them use crawl-66-249-66-1.googlebot.com, the name of the address Google's verification page uses as its first example. Here's what each of the four returns, with a documentation address standing in for theirs:

$ dig +short -x 203.0.113.7
crawl-66-249-66-1.googlebot.com.
$ dig +short crawl-66-249-66-1.googlebot.com
66.249.66.1

The forward lookup succeeds, and its answer is in Google's own list. A check that asks only whether the name resolves, or whether it resolves into Google's ranges, passes all ten. Google's method asks for more: verify that the answer is "the same as the original accessing IP address from your logs". That comparison fails all ten.

Bingbot's name is forged far less: outside Microsoft's networks, four addresses carry a search.msn.com name that fails the forward lookup, two that don't exist and two that copy a name of Microsoft's. No address outside Apple carries an Applebot name.

622 stale names on Googlebot's old block

The other 622 Googlebot failures are in four /24s on CenturyLink's network, AS3561: 64.68.88.0/24, 64.68.90.0/24, 64.68.91.0/24 and 64.68.92.0/24. Their reverse DNS, served by CenturyLink's own name servers, answers 18 names like crawl1.googlebot.com and crawler16.googlebot.com, none of which exists forward.

Googlebot crawled from this part of 64.68 in the early 2000s. A list of Google's crawler addresses kept at iplists.com, as archived in February 2004, has 64.68.80.0 to 64.68.88.255 and 160 addresses in 64.68.90, and in a WebmasterWorld thread from May 2003 a visit from crawler11.googlebot.com traces to 64.68.82. A check that trusts the reverse name alone still admits all 622 as Googlebot.

What each list leaves out

A confirmed name doesn't mean the address crawls today. It means the operator's DNS vouches for an address its list leaves out, so the operator's two checks give different answers for it.

Google's common-crawlers.json, the file for Googlebot, holds 170 IPv4 prefixes, all /27s and /28s (the old googlebot.json address now redirects there), while Google's DNS names whole /24s. In 66.249.67.0/24, 66.249.79.0/24 and 192.178.5.0/24, the /27s Google doesn't list hold 416 confirmed Googlebot names. The other 256 fill 203.208.60.0/24 on AS24424, which the regional registry, APNIC, lists as GOOGLECN, registered to Beijing Gu Xiang Information Technology with google.com contacts. In 2010 The Register called the company Google's website operator in China. That /24 is in none of Google's five crawler and fetcher files, nor in its general list of Google's addresses.

Bing's bingbot.json is dated January 3, 2024, and its 28 prefixes are the ones the Internet Archive captured in May 2024. Microsoft's DNS confirms Bingbot names in 185 /24s, and the list holds the names in 30 of them. The other 155 hold the 27,554, 26,545 of them on Microsoft's AS8075. Bing's page offers both checks, and asks for the list to be refreshed daily "as we can change this list any time".

Apple's applebot.json has changed twice this year. PPC Land reported in August that it grew from 12 prefixes to 33. The version dated September 15 has 24: it dropped nine /24s that the file had held since 2023, and Apple's DNS still names 1,929 addresses in them as Applebot, every one forward-confirmed.

Apple's own example is among the 3,246. About Applebot shows 17.58.101.179 resolving to 17-58-101-179.applebot.apple.com and back, and that address is in none of the three versions.

Google's domain check accepts any Google Cloud server

Step two of Google's method accepts a name that ends in googlebot.com, google.com or googleusercontent.com. Its table of crawler types gives each one its own pattern, and Googlebot's names are all under googlebot.com. The difference matters: googleusercontent.com is also the domain of the default reverse DNS name Google Cloud gives a VM's address. 20,128,093 IPv4 addresses had a bc.googleusercontent.com name in IP RDNS on October 16, and 409 of a random 412 forward-confirmed. A check that accepts all three domains accepts anyone renting a Google Cloud VM as Google.

A check that holds up

1. Match the address against the operator's current list. It's a local lookup, and a hit settles it. Fetch the files daily.
2. Otherwise, look up the address's reverse DNS name and require the crawler's own domain: googlebot.com for Googlebot, never google.com or googleusercontent.com.
3. Look that name up forward, and accept the address only if it's among the answers. "The name resolves" lets the ten copied names through, and so does "it resolves to a Google address".
4. Cache the answer by address, refusals included, and clear the cache when you fetch the lists.

In Python, with the standard library only:

import ipaddress
import json
import socket
from functools import lru_cache

# The name each operator gives its crawler in reverse DNS.
SUFFIX = {
    "googlebot": ".googlebot.com",
    "bingbot": ".search.msn.com",
    "applebot": ".applebot.apple.com",
}


def load_list(path):
    with open(path) as f:
        prefixes = json.load(f)["prefixes"]
    return [
        ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
        for p in prefixes
    ]


@lru_cache(maxsize=100_000)
def dns_confirms(ip, suffix):
    try:
        name = socket.gethostbyaddr(ip)[0].rstrip(".").lower()
        if not name.endswith(suffix):
            return False
        answers = socket.getaddrinfo(name, None)
    except OSError:
        return False
    addr = ipaddress.ip_address(ip)
    return any(ipaddress.ip_address(a[4][0]) == addr for a in answers)


def is_crawler(ip, crawler, listed):
    if any(ipaddress.ip_address(ip) in net for net in listed):
        return True
    return dns_confirms(ip, SUFFIX[crawler])

listed is the operator's file, loaded with load_list: common-crawlers.json for Googlebot. We ran it on October 16 on an address from each case above. It accepted listed addresses, confirmed ones outside each list and Apple's example, and refused the 28-address run, a copied name, a stale 64.68 name and a Google Cloud VM.

Checking every address at once

A check at request time only sees the addresses that visit you. With IP RDNS, the reverse half runs offline over every IPv4 address with a name: the names ending in a crawler's domain are every address whose reverse DNS claims it, 46,589 for these three crawlers on October 16, and only those need a forward lookup. Ours took about a minute. For a single address, our reverse DNS lookup shows the name IP RDNS holds for it.

IP ASN and ASN, both free with any account, give the network each address is on and what kind of network it is. Hosting IP has 75 of the 76 forged addresses as built on October 16, 63 of them at high confidence. A request that claims to be Googlebot from a hosting range that isn't Google's fits every forgery we found but one.

How we measured

The names are every IPv4 address in IP RDNS on October 16, 2026 whose PTR name ends in .googlebot.com, .search.msn.com or .applebot.apple.com. We looked each name up forward through an ordinary recursive resolver between 22:22 and 22:23 UTC that day, and counted an address as confirmed when the name's A records included it. We downloaded the lists from their official addresses at 22:26 UTC: Google's five crawler and fetcher files (common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json), Bing's bingbot.json and Apple's applebot.json. A Googlebot address counts as listed when any of Google's five files holds it; all 4,911 are in common-crawlers.json.

We looked up the reverse DNS of the 702 failing Googlebot and non-Microsoft Bingbot addresses again between 22:45 and 22:46 UTC, and every one answered the same name. The forward lookups that had returned NXDOMAIN did so again at 22:58 UTC, and every lookup printed here answered as shown at 23:08 UTC. Networks and their types are from IP ASN and ASN as built on October 16. The earlier versions of applebot.json are the Internet Archive's captures of October 28, 2025 and September 13, 2026.

IP RDNS is a gzipped CSV download, and its docs list the columns. To license it, see pricing or talk to us.