Skip to content
InternetData
Reference

Let the real crawlers in. Keep the impostors out.

Any client can send Googlebot's user agent, so a rule that lets crawlers in by name lets in everyone who copies it. These are the 98 bots Bot IP names, each with its operator and the way the operator lets you check that a request is really its own.

Check whose bot uses an address

Enter an IP address to open its page at the Bot IP card: whose bot uses it, the bots by name, what they do and whether their operator vouches for the address.

Every bot, by what it does

A published list or a documented reverse DNS name lets you prove a request is the operator's; a bot with none published can only be taken at its word.

Search engine

Indexes pages for a web search engine, such as Googlebot and Bingbot.

BOTOPERATORVERIFY BY
ApplebotApplePublished list
BaiduspiderBaiduReverse DNS
bingbotMicrosoftPublished list
DuckDuckBotDuckDuckGoPublished list
GooglebotGooglePublished list
MojeekBotMojeekPublished list
PetalBotHuaweiReverse DNS
SeznamBotSeznam.czPublished list
Sogou SpiderSogouNone published
Yahoo SlurpYahooReverse DNS
YandexBotYandexReverse DNS
YetiNaverPublished list

AI training

Collects pages to train AI models, such as GPTBot, ClaudeBot and CCBot.

BOTOPERATORVERIFY BY
Ai2BotAllen Institute for AINone published
AmazonbotAmazonPublished list
BytespiderByteDanceNone published
ClaudeBotAnthropicPublished list
cohere-aiCohereNone published
DiffbotDiffbotNone published
GPTBotOpenAIPublished list
ImagesiftBotImageSiftNone published
Meta-ExternalAgentMetaRegistered space
PanguBotHuaweiNone published
ReflectionbotReflection AIPublished list

AI assistant

Fetches a page because a person asked an AI assistant about it, such as ChatGPT-User and Claude-User.

BOTOPERATORVERIFY BY
Amzn-UserAmazonPublished list
ChatGPT-UserOpenAIPublished list
Claude-UserAnthropicPublished list
DuckAssistBotDuckDuckGoPublished list
Google-NotebookLMGooglePublished list
Keenable-UserKeenableNone published
Meta-ExternalFetcherMetaRegistered space
MistralAI-UserMistral AIPublished list
Perplexity-UserPerplexityPublished list

AI agent

Browses for a person, opening pages and filling in forms, such as ChatGPT agent and Google-Agent.

BOTOPERATORVERIFY BY
Google-AgentGooglePublished list

SEO

Audits sites and maps their links for SEO tools, such as AhrefsBot and SemrushBot.

BOTOPERATORVERIFY BY
AhrefsBotAhrefsPublished list
AhrefsSiteAuditAhrefsPublished list
BarkrowlerBabbarNone published
BLEXBotWebMeUpNone published
DataForSeoBotDataForSEOReverse DNS
DotBotMozNone published
LumarLumarPublished list
MJ12botMajesticNone published
SemrushBotSemrushNone published
SERankingBacklinksBotSE RankingNone published
serpstatbotSerpstatNone published

Ads

Checks ad landing pages and where ads appear, such as AdsBot-Google and TTD-Content.

BOTOPERATORVERIFY BY
AdsBot-GoogleGooglePublished list
IAS crawlerIntegral Ad SciencePublished list
OAI-AdsBotOpenAINone published
TTD-ContentThe Trade DeskPublished list

Link preview

Fetches a page to show its preview when someone shares the link, such as facebookexternalhit and Slackbot.

BOTOPERATORVERIFY BY
DiscordbotDiscordNone published
EmbedlyEmbedlyNone published
facebookexternalhitMetaRegistered space
FlipboardProxyFlipboardPublished list
LinkedInBotLinkedInNone published
MicrosoftPreviewMicrosoftPublished list
PinterestbotPinterestNone published
redditbotRedditNone published
SlackbotSlackNone published
TelegramBotTelegramPublished list
TwitterbotXNone published

Feed fetcher

Polls feeds for readers and aggregators, such as FeedFetcher-Google and FlipboardRSS.

BOTOPERATORVERIFY BY
FeedFetcher-GoogleGooglePublished list
FlipboardRSSFlipboardPublished list
RSSAPIRSS APIPublished list

Monitoring

Checks a site's uptime and speed for its owner, such as UptimeRobot and Pingdom.

BOTOPERATORVERIFY BY
Better Uptime BotBetter StackPublished list
Datadog SyntheticsDatadogPublished list
InstatusBotInstatusNone published
PingdomSolarWinds PingdomPublished list
Site24x7Site24x7None published
StatusCakeStatusCakePublished list
UptimeRobotUptimeRobotPublished list

Archiver

Saves copies of pages for a web archive, such as archive.org_bot and CCBot.

BOTOPERATORVERIFY BY
archive.org_botInternet ArchiveNone published
CCBotCommon CrawlPublished list

Internet scanner

Probes addresses across the whole internet for research and exposure data, such as Censys and Shodan.

BOTOPERATORVERIFY BY
BinaryEdgeBinaryEdgeNone published
CensysInspectCensysNone published
ExpansePalo Alto NetworksNone published
InternetDataInternetDataPublished list
InternetMeasurementinternet-measurement.comNone published
OnypheOnypheNone published
ShadowserverThe Shadowserver FoundationNone published
ShodanShodanNone published
StretchoidStretchoidNone published

Webhook

Sends event callbacks to the URLs a platform's customers register, such as Stripe and GitHub.

BOTOPERATORVERIFY BY
GitHub-HookshotGitHubPublished list
StripeStripePublished list

Other

Automation its operator names that no other category fits.

BOTOPERATORVERIFY BY
Google-Read-AloudGooglePublished list
Google-Site-VerificationGooglePublished list
GoogleOtherGooglePublished list
Parse.lyParse.lyPublished list

The same check in every lookup

The lookup API answers is_bot on every plan, the free one included. A paid plan adds bot, whose bot it is and whether its operator vouches for the address, and with a request's user agent sent beside the address, a verdict on it: verified, spoofed or unverifiable.

Lookup API · Pricing

00 · Unlimited

Unlimited lookups, no sign-up.

Paste it into a terminal. Any address answers with its country, network, company and hostname, with no key and no daily or monthly cap. ASNs and domains answer too.

curl https://api.internetdata.io/8.8.8.8
{
"ip": "8.8.8.8",
"is_bogon": false,
"is_tor": false,
"is_relay": false,
"is_bot": false,
"is_anycast": true,
"is_carrier": false,
"is_satellite": false,
"is_ssh": false,
"is_smtp": false,
"is_web": true,
"is_dns_resolver": true,
"is_bittorrent": false,
"hostname": "dns.google",
"location": {
"continent": "NA",
"continent_name": "North America",
"country": "US",
"country_name": "United States",
"timezone": "America/Los_Angeles",
"currency_code": "USD",
"currency_name": "Dollar",
"currency_symbol": "$",
"flag_emoji": "🇺🇸",
"flag_unicode": "U+1F1FA U+1F1F8"
},
"asn": {
"asn": 15169,
"name": "Google LLC",
"domain": "google.com",
"type": "hosting",
"prefix": "8.8.8.0/24",
"country": "US",
"rpki_status": "valid"
},
"company": {
"name": "Google LLC",
"domain": "google.com"
},
"tor": {},
"relay": {},
"abuse": {
"email": "network-abuse@google.com"
}
}
01 · Every field

All of the data on a paid plan.

VPN, proxy, hosting and CDN flags, each naming the operator.
Location to the city, with coordinates and an accuracy radius.
WHOIS records, the whole abuse contact and the services an address runs.
Up to 1,000 addresses, ASNs and domains in one request.

A free API key tries every field, with no card.