Analytics & TestingArtificial IntelligenceData and Development

Good Bots, Bad Bots, and AI Agents: How to Identify Who’s Really Visiting Your Site

Every few weeks, someone running a SaaS platform asks me why their server bill keeps climbing while their analytics look flat. The answer is almost always the same. A growing share of the requests hitting their infrastructure never shows up in GA4, and most of it is not human. Search crawlers, AI training crawlers, AI search indexers, assistants fetching a page because someone asked a question, and now full browsers driven by agents are all reading your site. So are the malicious bots that stuff credentials, steal content, and test stolen cards, often while pretending to be Googlebot or an ordinary Chrome user. Your analytics platform was never designed to tell these apart.

If you host your own application or content, you have something your analytics vendor does not: the raw request. That request carries a user agent, an IP address, headers, timing, and behavior. With a little discipline, you can identify most automated traffic, verify the bots that claim to be legitimate, and decide what to do about the rest. This article walks through what GA4 can and cannot show you, how to recognize malicious bots that spoof legitimate identities, why blocking everyone or allowing only search engines backfires, the bots and agents you should know by name, and working code to classify and verify them on your own servers.

Key Takeaways

  • GA4 silently excludes known bot traffic, and you cannot turn that off or see how much was removed. Most crawlers never run your tag, and GA4 has no user agent dimension.
  • Malicious bots stuff credentials, scrape content, hoard inventory, test stolen cards, commit ad fraud, spam forms, scan for vulnerabilities, flood your edge, and create fake accounts. They routinely spoof legitimate user agents, including Googlebot and ChatGPT-User.
  • A user agent is only a claim. Verify important bots with reverse DNS, published IP ranges, or Web Bot Auth signatures. A failed check is spoofed, not a good bot.
  • Blocking every bot, or allowing only search engines, cuts you out of AI search and assistants while doing little about attackers who already look like browsers. That is especially costly for SaaS vendors.
  • Classify by purpose, verify the claim, allow verified good bots per purpose (assistants and training can be decided separately), rate-limit or challenge unverified traffic, and block spoofed requests and bad behavior.
  • Your server logs, CDN logs, and application middleware are the only places that see every request. Agents driving real browsers can still look like people.

Why Bot Identification Is Now a Business Problem

Bots used to be a nuisance you handled with a few robots.txt lines and a firewall rule. That changed when AI companies began crawling the web at scale and when assistants began fetching pages on behalf of their users. The volume is no longer a rounding error, and the purpose of each request now matters. A training crawler may take your content and never send a visitor. An assistant fetcher may be the only way a buyer ever reads your pricing page.

Automated traffic accounted for more than 53% of all web traffic in 2025, up from 51% in 2024, and bad bots alone made up 40% of traffic. AI-enabled bot attacks grew 12.5 times, and 27% of bot attacks targeted APIs.

Imperva, a Thales company, 2026 Bad Bot Report

Across Cloudflare’s network in 2025, AI bots averaged 4.2% of HTML requests, while Googlebot alone accounted for 4.5%. Anthropic’s crawl-to-refer ratio peaked as high as 500,000 to 1, OpenAI’s reached 3,700 to 1, and Google’s ranged from about 3 to 30 to 1.

Cloudflare, Radar 2025 Year in Review, December 2025

For a SaaS vendor, that ratio is the story. Crawlers that take far more than they refer consume bandwidth, compute, and CDN egress you pay for. They hit expensive dynamic routes, search pages, and API endpoints. And because they rarely show up in analytics, nobody on the marketing team can see the cost or the opportunity. I have written before about how AI is breaking the buyer journey we thought we could measure; bot identification is the operational side of that same problem.

What GA4 Can and Cannot Tell You About Bots

Let me be precise here, because I see a lot of confusion about this. GA4 does not give you a bot report. It gives you a filter you cannot see, a tag most bots never run, and no field to identify them by.

GA4 Excludes Known Bots and Hides the Evidence

Google states that traffic from known bots and spiders is automatically excluded from GA4 properties. Google identifies that traffic using a combination of its own research and the IAB Tech Lab International Spiders and Bots List. The IAB list is a commercial reference used across ad verification to filter general invalid traffic (GIVT). Google is explicit that you cannot disable known bot traffic exclusion, and you cannot see how much known bot traffic was excluded.

That is a reasonable default for marketing reporting. It is useless for anyone who wants to understand automated load, AI crawler access, or agent behavior. The exclusion is a black box, its membership is not published by Google, and you get no count, no list, and no sample of what was removed.

Most Crawlers Never Run Your GA4 Tag

The bigger issue is upstream. GA4 collects data when the gtag.js or Google Tag Manager script executes in a browser. Crawlers that fetch your HTML and never execute JavaScript never fire the tag, so they never reach GA4 at all, excluded or not.

In an analysis of crawler traffic across the Vercel network, none of the major AI crawlers rendered JavaScript, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider, and PerplexityBot. GPTBot made 569 million fetches and Claude 370 million in a month, compared with 4.5 billion for Googlebot, and about 34% of fetches from both ChatGPT and Claude hit 404 pages.

Vercel and MERJ, The Rise of the AI Crawler, December 17, 2024

Googlebot and Applebot do render JavaScript, which is one reason how Googlebot sees your website matters for SEO. But even when a rendering crawler executes your tag, it is exactly the kind of traffic the known bot exclusion is built to remove.

There Is No User Agent Dimension

Even if a bot did execute your tag and slipped past the exclusion, GA4 would not tell you what it was. GA4 has no user agent dimension in its reports or explorations. The BigQuery export schema includes fields such as device.web_info.browser, browser_version, and hostname, but not the raw user agent string. GA4 data filters only cover developer traffic, internal traffic defined by IP address, and unwanted referrals. There is no filter by user agent.

The AI Assistant Channel Measures People, Not Bots

In 2026, Google added an AI Assistant channel to the GA4 default channel group. Google describes it as the channel by which users arrive from sources like ChatGPT, Gemini, Deepseek, Copilot, or Grok, and notes that it excludes Google’s AI Overviews and AI Mode. It is useful, but it is referrer-based. It measures people who clicked a citation inside an assistant and landed in a real browser that ran your tag. It does not measure the assistant’s fetcher that read your page to write the answer, and it says nothing about training crawlers.

What Is Partially Possible in GA4

There are three ways to get some bot visibility into the Google stack, and each has limits you should understand before you build anything.

  • Measurement Protocol: your server can classify each request and send a custom event to GA4 over HTTP. This is the most practical option, and I include a sample at the end. Send it to a separate GA4 property, never your marketing property, so you do not pollute sessions, users, and conversions. Keep the API secret on the server, and know the limits: 25 events per request, 25 parameters per event, and parameter values capped at 100 characters on standard properties. Google does not document how the known bot exclusion treats Measurement Protocol events, so validate with the debug endpoint and compare counts against your logs.
  • Server-side tagging: a server-side GTM container receives the user agent and IP in its common event data, so you can enrich or route events by user agent. The catch is that it only sees requests that reach it, which means browsers that ran your tag. It does not see crawlers that only fetched your HTML unless you forward those requests to it yourself.
  • BigQuery export: exporting GA4 to BigQuery gives you event-level data, but it is still only what GA4 collected. If you send Measurement Protocol bot events, BigQuery is where you can join them to log data. It will not recover what GA4 excluded or never saw.
Diagram comparing which bots and AI agents appear in server and CDN logs, run the GA4 tag, and show up in GA4 reports

The Agentic Blind Spot

This is the part that worries me. For twenty years, the analytics industry assumed a visit meant a person in a browser. Bots were noise to filter out. In an agentic world, automated requests are increasingly how your buyers actually consume your content.

Consider what happens when a buyer asks an assistant to compare three vendors. The assistant’s fetcher reads your pricing page, your documentation, and your security page. None of that runs your tag. The buyer may never visit. If they do, they arrive later as AI Assistant or Direct traffic, with no record of the research that happened on their behalf. Your analytics show a quiet week while your content did the selling.

Now add agents that drive real browsers. OpenAI’s cloud browser uses a standard Chrome user agent. Anthropic’s hosted browser adds a Claude-Agent token to an otherwise normal Chrome string. These agents execute JavaScript, so they can run your tag, accept or ignore your consent banner, and appear in GA4 as ordinary human sessions. Your conversion rates, engagement metrics, and audience reports start to mix people and software, and GA4 gives you no dimension to separate them.

That is a woeful position for anyone who makes decisions from analytics. The traffic that matters most in the next few years is exactly the traffic GA4 is designed to drop or cannot distinguish. If you want to understand how AI systems consume your site, you have to measure it where it happens: at your server and your edge.

Malicious Bots, and Why They Pretend to Be Someone Else

Not every automated request is a crawler with a published policy. A large, well-funded share of bot traffic is there to steal, defraud, or knock you over. These are the malicious bots I mean, and they are not a footnote. They are why so many companies now reach for a blunt block, and why that block keeps missing the traffic that actually hurts them.

Credential stuffing takes leaked username and password pairs and tries them on your login form or your identity API. Scraping and content theft copy pricing, inventory, reviews, or entire documentation sets, either to resell or to undercut you. Inventory hoarding and scalping reserve or buy limited stock faster than any person could. Card testing, sometimes called carding, runs stolen payment numbers through your checkout in small amounts to see which cards still work. Click and ad fraud burns your acquisition budget on fake impressions and fake conversions. Form and comment spam fills lead forms, sign-ups, and discussion threads with junk. Vulnerability scanning probes for known flaws, open admin paths, and misconfigured APIs. DDoS and layer-7 floods try to exhaust your application, not just your pipe. Fake account creation industrially opens accounts that are later used for abuse, referrals, or further fraud.

None of that requires an honest user agent. A stuffing script can say it is Chrome. A scraper can say it is Googlebot. An attacker who wants to slip past a naive allowlist can say it is ChatGPT-User. That is why the classification in this article is only the first step, and why verification matters more than matching a token.

Financial services accounted for 24% of all bot attacks and 46% of account takeover incidents in 2025, a reminder that a large share of malicious automation is aimed at identity systems and APIs, not just at public pages.

Imperva, a Thales company, 2026 Bad Bot Report, April 29, 2026

Across a 30-day sample in Akamai’s 2026 Fraud and Abuse report, 26.6% of recorded login attempts were high- or critical-risk, a level the firm associates with brute force and credential stuffing. Account verification requests made up 40% of that high- and critical-risk login traffic, and commerce accounted for 41.1% of all login attempts.

Akamai, AI-Driven Digital Deception: The Fraud and Abuse Report 2026, October 29, 2026

F5 Labs analyzed 207 billion web and API transactions from customers who already had bot defenses in place, from November 2023 to September 2024. Of those, 10.2% were automated and 4.8% were malicious bot traffic that was mitigated. On the web, 33.5% of login attempts at technology companies were attempted account takeovers, ahead of general retail at 25.7% and gaming at 19.6%.

F5 Labs, 2025 Advanced Persistent Bots Report, March 28, 2025

In the first half of 2026, Cloudflare mitigated 23.2 million network-layer DDoS attacks and 29.64 trillion HTTP DDoS requests, about 5,300 network-layer attacks every hour. Most of those attacks were short, and 90.60% ended in under 10 minutes, which is why automated defenses matter more than a human looking at a dashboard.

Cloudflare, DDoS Threat Report H1 2026, August 11, 2026

The last point is easy to miss. A request that claims to be Googlebot, Bingbot, or ChatGPT-User is asking you to trust it. If the IP is not in the operator’s published ranges, if reverse DNS plus a forward confirmation fails, or if a Web Bot Auth signature does not verify, that request is spoofed. It is not a good bot that failed a check. It is an impersonation, and you should treat it as malicious. The code later in this article returns a spoofed class for exactly that case.

Spoofing also works in the other direction. Many malicious clients never bother to impersonate a named crawler. They send a normal Chrome or Safari user agent and try to look like a person. Those requests land in the browser class until rate, path, honeypot, or challenge signals give them away. Verification catches the pretenders who borrow a famous name. Behavior catches the ones who borrow a browser.

Two Policies That Look Safe and Cost You Real Visitors

I keep seeing the same two reactions, especially from teams that just got a scary bot report or a rising CDN bill. Both feel responsible. Both can do more damage than the bots they were meant to stop.

Blocking Everything

The first policy is a blanket block. A WAF or bot-fight mode that challenges or drops anything that does not look like a person. A robots.txt file that disallows every user agent. A CDN toggle that turns off all AI crawlers in one click. On a bad week, I understand the impulse. It also hides you.

AI search crawlers cannot build the index that cites you. Assistant fetchers cannot read your pricing or docs when a buyer asks a question, so the answer is written from stale training data, a competitor’s page, or nothing at all. Link preview fetchers fail, so the URL someone pastes into Slack or Facebook looks like a broken card. And agents acting in a real browser for a real customer or prospect can be challenged, rate-limited, or blocked as if they were an attack. For a SaaS vendor, that last one is the expensive miss. Your buyers increasingly research, compare, and even start a trial through an assistant. If the assistant cannot reach you, you do not just lose a crawl. You lose the conversation.

A blanket block also gives you a false sense of safety. The malicious bots that already spoof a normal browser will often pass the same filter you just used to shut out OAI-SearchBot and ChatGPT-User. You paid for invisibility in AI answers, and you still have credential stuffing on the login page.

Allowing Only Search Engines

The second policy is more surgical and still wrong for most companies I work with: allow Googlebot, Bingbot, and maybe Applebot, and block everything else. You keep classic organic search, which still matters. You also shut out the AI search crawlers and assistant fetchers that now drive a growing share of discovery. OAI-SearchBot, Claude-SearchBot, PerplexityBot, and their user-initiated counterparts are not Googlebot, and they will not get a pass under a search-engines-only rule.

This policy does nothing about malicious bots that spoof browsers, because those requests never claim to be Googlebot in a way you can allowlist. It can even make spoofing more attractive: if the only automated traffic you trust is a short list of famous user agents, an attacker has a clear list of names to forge. Unless you verify those names, an allowlist is an invitation.

A Better Approach

Classify first, then verify, then apply a policy by purpose. That is the whole method, and it is what the code at the end of this article is built to do.

  • Classify the claim by purpose, using the ordered taxonomy below, so an assistant fetch is never filed as a training crawl.
  • Verify any request that claims to be a known bot or carries a Signature-Agent header. Use Web Bot Auth signatures where they exist, published IP ranges, or reverse DNS with a forward confirmation.
  • Allow verified good bots according to purpose. You can let assistants and AI search in, rate-limit SEO tools, and disallow training crawlers, or the reverse. Those are separate decisions.
  • Rate-limit or challenge unverified traffic, including requests that look like browsers. Do not pretend a Chrome user agent is a person.
  • Block spoofed claims and verified-bad behavior: credential stuffing, card testing, inventory hoarding, honeypot hits, and the rest of the list above.

That is how you stay visible to the agentic web without opening the door to whoever is willing to type Googlebot in a header. It also gives you a log you can defend: not we blocked the bots, but we allowed this verified assistant, we challenged this unknown browser, and we dropped this spoofed crawler.

A Taxonomy That Survives the Agentic Web

Before you write a single rule, decide what you are classifying for. I sort automated traffic by purpose, because purpose determines the business decision. Blocking a training crawler is a licensing choice. Blocking an assistant fetcher may cut you out of an answer a buyer is reading right now.

A bot taxonomy is an ordered list of traffic classes, each defined by why the request was made. Classifying by purpose, rather than by company, lets you set policy per class: allow, rate limit, block, or measure.

  1. assistant: a fetch or browser session triggered by a person, such as ChatGPT-User, Claude-User, Perplexity-User, or an agent in a hosted browser.
  2. ai_search: crawlers that build an index for AI answers and citations, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot.
  3. ai_training: crawlers that collect content for model training, such as GPTBot, ClaudeBot, Meta-ExternalAgent, and CCBot.
  4. search: traditional search engine crawlers, such as Googlebot, Bingbot, Applebot, and YandexBot.
  5. seo: SEO tool and inspection crawlers, such as AhrefsBot, SemrushBot, and Google-InspectionTool.
  6. preview: link preview fetchers in social and messaging apps, such as facebookexternalhit and Slackbot-LinkExpanding.
  7. bot: other identified bots, such as ad landing page checkers, feed fetchers, and anything with bot, crawler, or spider in its user agent.
  8. script: HTTP libraries and headless tools, such as curl, python-requests, Go-http-client, and HeadlessChrome.
  9. other: empty or unrecognized user agents.
  10. browser: everything that looks like a real browser engine. This is a default, not a verdict.

Order matters because the first match wins. Several companies give their person-triggered fetcher and their crawler similar names. If you test for ClaudeBot before Claude-User, or for a generic bot pattern before a specific token, you will misfile traffic. Check the assistant tokens first, then AI search, then training, and only fall back to generic patterns at the end.

Spoofed is not an eleventh user agent class. It is the outcome of verification. When a request claims a known identity or presents a Signature-Agent header, and every check you actually ran fails, the class becomes spoofed. The original name stays on the record, so you can see that it claimed to be GPTBot or Googlebot. Browser, script, and empty user agents are not marked spoofed just because they were not in a crawler’s IP list; those still need rate and behavior signals.

Bot classification flow: ten ordered user agent classes where the first match wins, then verify the claim with Web Bot Auth, IP ranges or reverse DNS

Three caveats apply to everything that follows. First, a user agent is only a claim; anyone can send Googlebot in a header. Second, a request that carries a Signature-Agent header is announcing that it can be cryptographically verified, which is a far stronger signal than a user agent, but only once you verify the signature. Third, agents driving real browsers look like browsers, so the browser class will always contain some software.

The Known Bots and AI Agents

Here is the list I work from. Every token below comes from the operator’s own documentation, and I note where an operator does not publish one. Tokens are case-insensitive substrings of the user agent unless noted. Treat the robots.txt column as the operator’s stated policy, not a guarantee.

BotOperatorUser Agent TokenPurposeHonors robots.txtVerification
ChatGPT-UserOpenAIChatGPT-Userassistant: fetches when a ChatGPT user or GPT asksOpenAI says robots.txt rules may not applyPublished IP list
ChatGPT agent (cloud browser)OpenAIStandard Chrome user agentassistant: agentic browsing and commerce tasksActs for a userWeb Bot Auth signature, Signature-Agent chatgpt.com
Claude-UserAnthropicClaude-Userassistant: fetches when a Claude user asksYesPublished IP list
Claude-AgentAnthropicClaude-Agentassistant: hosted browser acting for a userManaged separately from crawlersPublished IP list (SAIL file)
Perplexity-UserPerplexityPerplexity-Userassistant: fetches when a user asksGenerally ignores robots.txtPublished IP list
MistralAI-UserMistral AIMistralAI-Userassistant: fetches for user questions in VibeControlled by its robots.txt tokenPublished IP list
Google-AgentGoogleGoogle-Agentassistant: agent fetches for Google productsUser-triggered fetchers generally ignore itPublished IP list; testing Web Bot Auth
Meta-ExternalFetcherMetameta-externalfetcherassistant: user-requested and agentic fetchesMay bypass robots.txtUser agent; no IP list on Meta’s crawler page
Amzn-UserAmazonAmzn-Userassistant: fetches for user actions such as AlexaMay not follow all directivesPublished IP list
DuckAssistBotDuckDuckGoDuckAssistBotassistant: real-time fetches for AI answers, not trainingYesPublished IP list
OAI-SearchBotOpenAIOAI-SearchBotai_search: ChatGPT search resultsYesPublished IP list
Claude-SearchBotAnthropicClaude-SearchBotai_search: improves Claude search resultsYes, including Crawl-delayPublished IP list
PerplexityBotPerplexityPerplexityBotai_search: Perplexity search index, not trainingYesPublished IP list
MistralAI-IndexMistral AIMistralAI-Indexai_search: Mistral search index, not trainingControlled by its robots.txt tokenPublished IP list
Amzn-SearchBotAmazonAmzn-SearchBotai_search: search and Alexa, not trainingYesPublished IP list
Meta-WebIndexerMetameta-webindexerai_search: Meta AI search and citationsYesUser agent; no IP list on Meta’s crawler page
GPTBotOpenAIGPTBotai_training: model trainingYesPublished IP list
ClaudeBotAnthropicClaudeBotai_training: model trainingYes, including Crawl-delayPublished IP list
Meta-ExternalAgentMetameta-externalagentai_training: AI training and indexingYesUser agent; no IP list on Meta’s crawler page
MistralAI-TrainingMistral AIMistralAI-Trainingai_training: model training datasetsYes, disallow it in robots.txtUser agent only
AmazonbotAmazonAmazonbotai_training: may be used to train Amazon AI modelsYes, but not Crawl-delayPublished IP list
CCBotCommon CrawlCCBotai_training: open web archive widely used for trainingYesReverse DNS and published IP list
BytespiderByteDanceBytespiderai_training: crawling attributed to ByteDanceNo official documentation; widely reported to ignore itNone published; enforce at the edge
Google-ExtendedGoogleNone, robots.txt token onlyControls Gemini training and grounding use of Googlebot crawlsIs itself a robots.txt controlNot applicable
Applebot-ExtendedAppleNone, robots.txt token onlyControls Apple AI training use of Applebot crawlsIs itself a robots.txt controlNot applicable
GooglebotGoogleGooglebotsearch: Google Search, including smartphone and desktopYes, always for automatic crawlsReverse DNS and published IP list
Storebot-GoogleGoogleStorebot-Googlesearch: Google Shopping surfacesYesReverse DNS and published IP list
BingbotMicrosoftbingbotsearch: Bing searchYesReverse DNS and published IP list
ApplebotAppleApplebotsearch: Siri and SpotlightYes, but not Crawl-delayReverse DNS and published IP list
YandexBotYandexYandexBotsearch: YandexSome Yandex robots ignore User-agent: *Reverse DNS
Google-InspectionToolGoogleGoogle-InspectionToolseo: Search Console testing toolsYesReverse DNS and published IP list
AhrefsBot and AhrefsSiteAuditAhrefsAhrefsBot, AhrefsSiteAuditseo: link index and site auditsYes, verified owners can let audits bypass itReverse DNS and published IP list
SemrushBot and SiteAuditBotSemrushSemrushBot, SiteAuditBotseo: backlink, audit, and other tool crawlersYes, Crawl-delay up to 10 secondsUser agent; Semrush advises against IP blocking
facebookexternalhitMetafacebookexternalhitpreview: link previews when content is sharedMay bypass for security and integrity checksUser agent; Meta recommends allowlisting its IPs
Slackbot-LinkExpandingSlackSlackbot-LinkExpandingpreview: link unfurls in SlackNo, Slack does not honor robots.txtUser agent
OAI-AdsBotOpenAIOAI-AdsBotbot: validates ad landing pagesNot statedPublished IP list
GoogleOther and Google-CloudVertexBotGoogleGoogleOther, Google-CloudVertexBotbot: internal research and Vertex AI agent crawls for site ownersYesReverse DNS and published IP list
FeedFetcher-GoogleGoogleFeedFetcher-Googlebot: RSS and Atom feedsUser-triggered; generally ignores itReverse DNS and published IP list

Assistant Fetchers and Agents

OpenAI documents ChatGPT-User as the agent that visits pages when a user or a custom GPT asks, and says robots.txt rules may not apply because the request is user-initiated. Its user agent looks like Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot. The newer ChatGPT agent runs a cloud browser that sends a standard Chrome user agent. OpenAI signs those requests with Web Bot Auth, using the Signature, Signature-Input, and Signature-Agent headers, with Signature-Agent set to “https://chatgpt.com”. OpenAI notes that Akamai, Cloudflare, HUMAN, and Vercel verify those signatures automatically.

Anthropic runs Claude-User for user-initiated fetches and says its bots respect robots.txt and Crawl-delay and will not bypass a CAPTCHA. Separately, Anthropic-hosted browsers used by products such as Claude Duet add a Claude-Agent token to a real Chrome user agent and publish their IP addresses in a SAIL file at claude.com/.well-known/sail. Anthropic is clear that anyone can send that header, so check the IP before you trust it.

Perplexity says Perplexity-User generally ignores robots.txt, since a person requested the page. Mistral runs MistralAI-User for user questions in its Vibe assistant. Google lists user-triggered fetchers such as Google-Agent, Google-Read-Aloud, GoogleMessages, and FeedFetcher-Google, and says these generally ignore robots.txt. Google-Agent publishes its own IP list, and Google says it is experimenting with Web Bot Auth for it. Meta uses meta-externalfetcher for user-requested and agentic fetches, Amazon uses Amzn-User, and DuckDuckGo uses DuckAssistBot for real-time fetches that feed its AI answers and are not used for training.

AI Search Crawlers

These build the indexes that AI answers cite. OAI-SearchBot surfaces sites in ChatGPT search and may append a robots.txt marker to its user agent when it fetches your robots file. Claude-SearchBot, PerplexityBot, MistralAI-Index, Amzn-SearchBot, and Meta-WebIndexer play the same role for their companies. Each operator says its search crawler is not used for model training. If you want to be cited in AI answers, this is the class you most likely want to allow.

AI Training Crawlers

GPTBot, ClaudeBot, Meta-ExternalAgent, MistralAI-Training, and Amazonbot collect content that may be used to train models. Common Crawl operates CCBot, whose open archive is widely used in training datasets, and it warns that fake CCBots exist, so verify by reverse DNS to crawl.commoncrawl.org or its published ranges. ByteDance publishes no official documentation for Bytespider that I could find, and it is widely reported to ignore robots.txt, so enforce any decision at your server, CDN, or WAF.

Google-Extended and Applebot-Extended are not crawlers at all. They are robots.txt tokens. Disallowing Google-Extended tells Google not to use content crawled by Googlebot for Gemini training and grounding, without affecting Search. Apple documents Applebot-Extended the same way for its AI models. You will never see either in a log. Anthropic also began honoring the emerging IETF AI preferences training signal in 2026, and Google has said it intends to support it.

Search Engine Crawlers

Googlebot always obeys robots.txt for automatic crawls and comes from hosts under googlebot.com. Bingbot resolves to search.msn.com, and Microsoft recommends refreshing its published IP list daily. Applebot resolves to applebot.apple.com, follows Googlebot rules when it is not named, and ignores Crawl-delay. Yandex verifies by reverse DNS to yandex.ru, yandex.net, or yandex.com, does not publish an IP list, and notes that some of its robots ignore User-agent: * rules. If you block these, you lose organic search, so verify before you act. My guide to writing a robots.txt file covers the directives.

SEO Tools, Previews, and Everything Else

Ahrefs runs AhrefsBot for its link index and AhrefsSiteAudit for audits, honors robots.txt and Crawl-delay, and publishes reverse DNS and IP ranges. Semrush runs SemrushBot plus a family of tool crawlers such as SiteAuditBot, honors robots.txt with a Crawl-delay of up to 10 seconds, and asks that you control it through robots.txt rather than IP blocking. Link preview fetchers such as facebookexternalhit and Slack unfurls exist so your links look good when shared; Slack states it does not honor robots.txt. Script clients such as curl, python-requests, and Go-http-client are not inherently malicious, since your own monitoring and integrations use them, but they deserve rate limits on expensive routes.

How to Verify a Bot Is Who It Claims to Be

Spoofing a user agent takes one line of code, so a good-bot allowlist based on user agents alone is an open door. I use three verification methods, from strongest to weakest, and I already covered blocking bots at the web server in an earlier article.

Web Bot Auth Signatures

Web Bot Auth is an IETF draft that lets a bot sign its HTTP requests using HTTP Message Signatures (RFC 9421). The request carries Signature, Signature-Input, and Signature-Agent headers, and the operator publishes its public keys in a directory at /.well-known/http-message-signatures-directory.

Signatures are the strongest proof available because they are cryptographic, survive shared cloud IPs, and work for agents in real browsers. The current IETF draft builds on RFC 9421. Formats are still evolving: OpenAI sends Signature-Agent as a quoted string, while the latest draft defines it as a dictionary. Never fetch keys from whatever host the header names. Pin the directories of operators you trust, cache their keys, and verify the signature over the covered components. If you are not ready to verify signatures yourself, your CDN may do it for you.

Published IP Ranges

OpenAI, Anthropic, Perplexity, Google, Bing, Apple, Amazon, Common Crawl, Mistral, DuckDuckGo, and Ahrefs publish JSON files of CIDR ranges for their bots. Download them on a schedule, cache them, and check the client IP against the list that matches the claimed user agent. A request that says GPTBot from an IP outside gptbot.json is not GPTBot. Support both IPv4 and IPv6.

Reverse DNS With Forward Confirmation

For Googlebot, Bingbot, Applebot, CCBot, YandexBot, and Ahrefs, you can do a reverse DNS lookup on the IP, using the PTR record, confirm that the hostname ends with the operator’s domain, then resolve that hostname forward and confirm it returns the same IP. Google documents this process for its crawlers and fetchers. The forward step matters, because anyone who controls the reverse zone for their own IP can make it claim to be googlebot.com. Cache results; do not do two DNS lookups on every request.

Behavior, Rate, Honeypots, and JavaScript Challenges

For everything without a published identity, behavior is your signal. Request rate per IP and per network, hits to URLs no human would click, a missing or inconsistent Accept-Language header, no cookies across many pages, and robots.txt fetches followed by disallowed paths all add up. A honeypot is a link hidden from people and disallowed in robots.txt; anything that requests it is automated and ignoring your rules. A JavaScript challenge separates clients that execute scripts from those that do not, which catches simple scrapers, but agents in real browsers pass it by design.

Let Your CDN Help

Cloudflare maintains a verified bots program that accepts Web Bot Auth, published IP lists with a stable user agent, or reverse DNS. In 2026 it reorganized its bot categories around purposes such as search, agent, training, transact, SEO, and social preview, and added signed agents to verification, along with AI crawler and agent access policies. Akamai, HUMAN, and Vercel also verify OpenAI’s signed agent traffic. Even with a CDN, keep your own classification in your logs, because your CDN’s dashboard is not your data warehouse.

Where to Capture Bot Traffic

You have four places to capture this, and I recommend at least two of them.

  • Web server access logs: nginx can classify every request with a map block and write the class into a dedicated log, with no application code. This sees everything that reaches your origin.
  • CDN or edge logs: these see requests your cache answered, which your origin never does. For a busy content site, most crawler hits may never reach the origin at all.
  • Application middleware: classify once per request in PHP, Node, Python, or Go, then use the result for rate limits, access rules, cache keys, and logging.
  • Analytics: optionally, send a summarized event to a separate GA4 property with the Measurement Protocol so marketers can see trends next to their normal reports.

Log the class, the bot name, the verification result, the path, the status code, and the bytes sent. That last one lets you put a cost on each class. Keep an eye on privacy: IP addresses are personal data in many jurisdictions, so set retention limits on raw logs and aggregate what you keep long term.

Code to Classify Bots on Your Own Servers

Here is a working implementation in the languages I see most often on SaaS and self-hosted stacks. Every sample reads the same ordered rules file, so you maintain one list. Each one lowercases the user agent, walks the rules in order, and returns the first match. If nothing matches, it checks for generic bot words, then for a Signature-Agent header, then for a browser engine, and finally falls back to other. Classification is the claim. Each language also has a classify_and_verify step (ClassifyAndVerify in Go) that keeps the claimed class when a check passes and returns spoofed when every check you actually ran has failed. Pass only the checks you performed: a Web Bot Auth result, an IP range match, or a reverse DNS result. Do not mark a normal browser as spoofed just because it was not in gptbot.json.

The Shared Rules File

Save this as bot-rules.json. Each entry is the class, a display name, and a lowercase token. Order is the policy: assistant tokens come first so Claude-User is never filed as ClaudeBot.

[
  ["assistant","ChatGPT-User","chatgpt-user"],
  ["assistant","Claude-User","claude-user"],
  ["assistant","Claude-Agent","claude-agent"],
  ["assistant","Perplexity-User","perplexity-user"],
  ["assistant","MistralAI-User","mistralai-user"],
  ["assistant","Google-Agent","google-agent"],
  ["assistant","Gemini Notebook","google-gemininotebook"],
  ["assistant","Google Read Aloud","google-read-aloud"],
  ["assistant","Meta-ExternalFetcher","meta-externalfetcher"],
  ["assistant","Amzn-User","amzn-user"],
  ["assistant","DuckAssistBot","duckassistbot"],
  ["ai_search","OAI-SearchBot","oai-searchbot"],
  ["ai_search","Claude-SearchBot","claude-searchbot"],
  ["ai_search","PerplexityBot","perplexitybot"],
  ["ai_search","MistralAI-Index","mistralai-index"],
  ["ai_search","Amzn-SearchBot","amzn-searchbot"],
  ["ai_search","Meta-WebIndexer","meta-webindexer"],
  ["ai_training","GPTBot","gptbot"],
  ["ai_training","ClaudeBot","claudebot"],
  ["ai_training","Meta-ExternalAgent","meta-externalagent"],
  ["ai_training","MistralAI-Training","mistralai-training"],
  ["ai_training","Amazonbot","amazonbot"],
  ["ai_training","CCBot","ccbot"],
  ["ai_training","Bytespider","bytespider"],
  ["search","Googlebot","googlebot"],
  ["search","Storebot-Google","storebot-google"],
  ["search","Bingbot","bingbot"],
  ["search","Applebot","applebot"],
  ["search","YandexBot","yandexbot"],
  ["seo","Google-InspectionTool","google-inspectiontool"],
  ["seo","AhrefsSiteAudit","ahrefssiteaudit"],
  ["seo","AhrefsBot","ahrefsbot"],
  ["seo","SemrushBot","semrushbot"],
  ["seo","SiteAuditBot","siteauditbot"],
  ["preview","facebookexternalhit","facebookexternalhit"],
  ["preview","Slackbot-LinkExpanding","slackbot-linkexpanding"],
  ["preview","Twitterbot","twitterbot"],
  ["preview","LinkedInBot","linkedinbot"],
  ["preview","Discordbot","discordbot"],
  ["preview","WhatsApp","whatsapp/"],
  ["preview","TelegramBot","telegrambot"],
  ["bot","Slackbot","slackbot"],
  ["bot","Meta-ExternalAds","meta-externalads"],
  ["bot","OAI-AdsBot","oai-adsbot"],
  ["bot","AdsBot-Google","adsbot-google"],
  ["bot","GoogleOther","googleother"],
  ["bot","Google-CloudVertexBot","google-cloudvertexbot"],
  ["bot","FeedFetcher-Google","feedfetcher-google"],
  ["script","curl","curl/"],
  ["script","Wget","wget/"],
  ["script","python-requests","python-requests"],
  ["script","python-httpx","python-httpx"],
  ["script","aiohttp","aiohttp"],
  ["script","Go-http-client","go-http-client"],
  ["script","node-fetch","node-fetch"],
  ["script","axios","axios/"],
  ["script","undici","undici"],
  ["script","okhttp","okhttp"],
  ["script","Java","java/"],
  ["script","Scrapy","scrapy"],
  ["script","HeadlessChrome","headlesschrome"]
]

PHP

This works in PHP 8.0 or later, in WordPress, Laravel, or plain PHP. The rules load once per process.

<?php
// bot-rules.json is an ordered list of [class, name, token]. First match wins.
function load_bot_rules(string $path = __DIR__ . '/bot-rules.json'): array
{
    static $rules = null;
    return $rules ??= json_decode(file_get_contents($path), true, flags: JSON_THROW_ON_ERROR);
}

function classify_request(string $ua, array $headers = []): array
{
    $u = strtolower(trim($ua));
    $h = array_change_key_case($headers, CASE_LOWER);
    $sig = $h['signature-agent'] ?? null;

    if ($u !== '') {
        foreach (load_bot_rules() as [$class, $name, $token]) {
            if (str_contains($u, $token)) {
                return ['class' => $class, 'name' => $name, 'signature_agent' => $sig];
            }
        }
        if (preg_match('/bot\b|crawler|spider/', $u)) {
            return ['class' => 'bot', 'name' => 'Generic bot', 'signature_agent' => $sig];
        }
    }
    $browser = str_contains($u, 'mozilla/') && preg_match('/applewebkit|gecko|trident/', $u);
    if ($sig !== null) {
        // A claim, not proof, until the RFC 9421 signature verifies.
        return ['class' => 'assistant', 'name' => 'Signed agent (unverified)', 'signature_agent' => $sig];
    }
    if ($browser) {
        return ['class' => 'browser', 'name' => 'Browser', 'signature_agent' => null];
    }
    return ['class' => 'other', 'name' => $u === '' ? 'Empty' : 'Unknown', 'signature_agent' => null];
}


// After you classify the claim, verify it. Pass only checks you actually ran.
// signature_ok: Web Bot Auth signature verified; in_ranges: IP in the operator list;
// rdns_ok: reverse DNS plus forward confirmation. Any true check verifies the claim.
// If every check you ran failed, the request is spoofed (a malicious impersonation).
function classify_and_verify(string $ua, array $headers = [], array $checks = []): array
{
    $r = classify_request($ua, $headers);
    $r['claimed_class'] = $r['class'];
    $r['verified'] = null;
    $named = !in_array($r['class'], ['browser', 'other', 'script'], true) && $r['name'] !== 'Generic bot';
    $signed = $r['signature_agent'] !== null && $r['signature_agent'] !== '';
    if (!$named && !$signed) {
        return $r;
    }
    $ran = [];
    foreach (['signature_ok', 'in_ranges', 'rdns_ok'] as $k) {
        if (array_key_exists($k, $checks) && $checks[$k] !== null) {
            $ran[] = (bool) $checks[$k];
        }
    }
    if (!$ran) {
        return $r;
    }
    if (in_array(true, $ran, true)) {
        $r['verified'] = true;
        return $r;
    }
    $r['verified'] = false;
    $r['class'] = 'spoofed';
    return $r;
}

// Usage: $r = classify_and_verify($_SERVER['HTTP_USER_AGENT'] ?? '', getallheaders(), ['in_ranges' => $ok]);

Node.js and Express Middleware

A Node.js 18 module with an Express middleware that classifies each request once and exposes the result to routes and logs.

// classify.js (Node 18+, CommonJS). bot-rules.json: ordered [class, name, token].
const RULES = require('./bot-rules.json');

function classifyRequest(ua = '', headers = {}) {
  const u = String(ua).trim().toLowerCase();
  const sig = headers['signature-agent'] ?? null; // Node lowercases header names
  if (u) {
    for (const [cls, name, token] of RULES) {
      if (u.includes(token)) return { class: cls, name, signatureAgent: sig };
    }
    if (/bot\b|crawler|spider/.test(u)) return { class: 'bot', name: 'Generic bot', signatureAgent: sig };
  }
  const browser = u.includes('mozilla/') && /applewebkit|gecko|trident/.test(u);
  if (sig) return { class: 'assistant', name: 'Signed agent (unverified)', signatureAgent: sig };
  if (browser) return { class: 'browser', name: 'Browser', signatureAgent: null };
  return { class: 'other', name: u ? 'Unknown' : 'Empty', signatureAgent: null };
}

// Express middleware: classify once, expose it to routes and logs.
function botClassifier(req, res, next) {
  req.botClass = classifyRequest(req.get('user-agent') || '', req.headers);
  res.set('X-Bot-Class', req.botClass.class); // optional, handy in CDN logs
  next();
}


// After you classify the claim, verify it. Pass only checks you actually ran.
// signatureOk: Web Bot Auth signature verified; inRanges: IP in the operator list;
// rdnsOk: reverse DNS plus forward confirmation. Any true check verifies the claim.
// If every check you ran failed, the request is spoofed (a malicious impersonation).
function classifyAndVerify(ua, headers = {}, checks = {}) {
  const r = { ...classifyRequest(ua, headers), verified: null };
  r.claimedClass = r.class;
  const named = !['browser', 'other', 'script'].includes(r.class) && r.name !== 'Generic bot';
  const signed = !!r.signatureAgent;
  if (!named && !signed) return r;
  const ran = ['signatureOk', 'inRanges', 'rdnsOk']
    .filter((k) => checks[k] !== undefined && checks[k] !== null)
    .map((k) => Boolean(checks[k]));
  if (!ran.length) return r;
  if (ran.some(Boolean)) { r.verified = true; return r; }
  r.verified = false;
  r.class = 'spoofed';
  return r;
}

module.exports = { classifyRequest, classifyAndVerify, botClassifier };
const express = require('express');
const { botClassifier } = require('./classify');

const app = express();
app.use(botClassifier);
app.get('/', (req, res) => res.json(req.botClass));
app.listen(3000);

Python

For Python 3.10 or later. It drops into Flask, Django, or FastAPI middleware.

# classify.py (Python 3.10+). bot-rules.json: ordered [class, name, token].
import json
import re
from pathlib import Path

RULES = json.loads((Path(__file__).parent / "bot-rules.json").read_text())
GENERIC = re.compile(r"bot\b|crawler|spider")
ENGINE = re.compile(r"applewebkit|gecko|trident")


def classify_request(ua: str = "", headers: dict | None = None) -> dict:
    u = (ua or "").strip().lower()
    h = {k.lower(): v for k, v in (headers or {}).items()}
    sig = h.get("signature-agent")
    if u:
        for cls, name, token in RULES:
            if token in u:
                return {"class": cls, "name": name, "signature_agent": sig}
        if GENERIC.search(u):
            return {"class": "bot", "name": "Generic bot", "signature_agent": sig}
    browser = "mozilla/" in u and bool(ENGINE.search(u))
    if sig:
        return {"class": "assistant", "name": "Signed agent (unverified)", "signature_agent": sig}
    if browser:
        return {"class": "browser", "name": "Browser", "signature_agent": None}
    return {"class": "other", "name": "Unknown" if u else "Empty", "signature_agent": None}



def classify_and_verify(ua: str = "", headers: dict | None = None, checks: dict | None = None) -> dict:
    """Verify a classified claim. Pass only checks you actually ran.

    signature_ok: Web Bot Auth signature verified; in_ranges: IP in the operator list;
    rdns_ok: reverse DNS plus forward confirmation. Any true check verifies the claim.
    If every check you ran failed, the request is spoofed (a malicious impersonation).
    """
    r = {**classify_request(ua, headers), "verified": None}
    r["claimed_class"] = r["class"]
    named = r["class"] not in {"browser", "other", "script"} and r["name"] != "Generic bot"
    signed = bool(r.get("signature_agent"))
    if not named and not signed:
        return r
    ran = [
        bool(checks[k])
        for k in ("signature_ok", "in_ranges", "rdns_ok")
        if checks and k in checks and checks[k] is not None
    ]
    if not ran:
        return r
    if any(ran):
        r["verified"] = True
        return r
    r["verified"] = False
    r["class"] = "spoofed"
    return r

Go

A Go package that embeds the rules file at build time and provides net/http middleware. Replace example.com/botclass with your own module path.

// Package botclass classifies requests with the shared, ordered bot-rules.json.
package botclass

import (
	"context"
	_ "embed"
	"encoding/json"
	"net/http"
	"regexp"
	"strings"
)

//go:embed bot-rules.json
var rulesJSON []byte

type Result struct {
	Class          string `json:"class"`
	Name           string `json:"name"`
	SignatureAgent string `json:"signature_agent,omitempty"`
	ClaimedClass   string `json:"claimed_class,omitempty"`
	// Verified is nil when no check was run, or a bool when it was.
	Verified *bool `json:"verified,omitempty"`
}

// Checks are the verification results you actually ran. Leave a field nil to skip it.
type Checks struct {
	SignatureOK *bool // Web Bot Auth signature verified
	InRanges    *bool // IP in the operator's published list
	RDNSOK      *bool // reverse DNS plus forward confirmation
}

var (
	rules   [][3]string // [class, name, token], first match wins
	generic = regexp.MustCompile(`bot\b|crawler|spider`)
	engine  = regexp.MustCompile(`applewebkit|gecko|trident`)
)

func init() {
	if err := json.Unmarshal(rulesJSON, &rules); err != nil {
		panic(err)
	}
}

func Classify(ua string, h http.Header) Result {
	u := strings.ToLower(strings.TrimSpace(ua))
	sig := h.Get("Signature-Agent")
	if u != "" {
		for _, r := range rules {
			if strings.Contains(u, r[2]) {
				return Result{Class: r[0], Name: r[1], SignatureAgent: sig}
			}
		}
		if generic.MatchString(u) {
			return Result{Class: "bot", Name: "Generic bot", SignatureAgent: sig}
		}
	}
	browser := strings.Contains(u, "mozilla/") && engine.MatchString(u)
	switch {
	case sig != "":
		return Result{Class: "assistant", Name: "Signed agent (unverified)", SignatureAgent: sig}
	case browser:
		return Result{Class: "browser", Name: "Browser"}
	case u == "":
		return Result{Class: "other", Name: "Empty"}
	}
	return Result{Class: "other", Name: "Unknown"}
}

// ClassifyAndVerify classifies the claim, then applies any checks you actually ran.
// Any true check verifies the claim. If every check you ran failed, Class becomes "spoofed".
func ClassifyAndVerify(ua string, h http.Header, c Checks) Result {
	r := Classify(ua, h)
	claimed := r.Class
	r.ClaimedClass = claimed
	named := claimed != "browser" && claimed != "other" && claimed != "script" && r.Name != "Generic bot"
	signed := r.SignatureAgent != ""
	if !named && !signed {
		return r
	}
	var ran []bool
	for _, v := range []*bool{c.SignatureOK, c.InRanges, c.RDNSOK} {
		if v != nil {
			ran = append(ran, *v)
		}
	}
	if len(ran) == 0 {
		return r
	}
	ok := false
	for _, v := range ran {
		if v {
			ok = true
			break
		}
	}
	r.Verified = &ok
	if !ok {
		r.Class = "spoofed"
	}
	return r
}

type ctxKey struct{}

// Middleware stores the result on the request context.
func Middleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		res := Classify(r.UserAgent(), r.Header)
		w.Header().Set("X-Bot-Class", res.Class)
		next.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), ctxKey{}, res)))
	})
}

func FromContext(ctx context.Context) (Result, bool) {
	res, ok := ctx.Value(ctxKey{}).(Result)
	return res, ok
}
// main.go: wire the middleware into any net/http server.
package main

import (
	"encoding/json"
	"log"
	"net/http"

	"example.com/botclass"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		res, _ := botclass.FromContext(r.Context())
		log.Printf("%s %s class=%s name=%q", r.RemoteAddr, r.URL.Path, res.Class, res.Name)
		json.NewEncoder(w).Encode(res)
	})
	log.Fatal(http.ListenAndServe(":8080", botclass.Middleware(mux)))
}

nginx Map and Log Format

If you want classification without touching your application, nginx can do it in the http block. Regex entries in a map are checked in order, so the first match wins here too. This writes a dedicated bot log with the class and whether a Signature-Agent header was present. Verification, and the spoofed outcome, still belong in your application or in a small service that can look up IP ranges and reverse DNS.

# Classification is only the claim. Verify published IP ranges or reverse DNS
# in your application (see verify.py). If the claim fails, set class=spoofed.
# http {} context. Regex entries are checked in order; first match wins.
map $http_user_agent $bot_class {
    default                      other;
    ""                           other;
    ~*chatgpt-user                 assistant;
    ~*claude-user                  assistant;
    ~*claude-agent                 assistant;
    ~*perplexity-user              assistant;
    ~*mistralai-user               assistant;
    ~*google-agent                 assistant;
    ~*google-gemininotebook        assistant;
    ~*google-read-aloud            assistant;
    ~*meta-externalfetcher         assistant;
    ~*amzn-user                    assistant;
    ~*duckassistbot                assistant;
    ~*oai-searchbot                ai_search;
    ~*claude-searchbot             ai_search;
    ~*perplexitybot                ai_search;
    ~*mistralai-index              ai_search;
    ~*amzn-searchbot               ai_search;
    ~*meta-webindexer              ai_search;
    ~*gptbot                       ai_training;
    ~*claudebot                    ai_training;
    ~*meta-externalagent           ai_training;
    ~*mistralai-training           ai_training;
    ~*amazonbot                    ai_training;
    ~*ccbot                        ai_training;
    ~*bytespider                   ai_training;
    ~*googlebot                    search;
    ~*storebot-google              search;
    ~*bingbot                      search;
    ~*applebot                     search;
    ~*yandexbot                    search;
    ~*google-inspectiontool        seo;
    ~*ahrefssiteaudit              seo;
    ~*ahrefsbot                    seo;
    ~*semrushbot                   seo;
    ~*siteauditbot                 seo;
    ~*facebookexternalhit          preview;
    ~*slackbot-linkexpanding       preview;
    ~*twitterbot                   preview;
    ~*linkedinbot                  preview;
    ~*discordbot                   preview;
    ~*whatsapp/                    preview;
    ~*telegrambot                  preview;
    ~*slackbot                     bot;
    ~*meta-externalads             bot;
    ~*oai-adsbot                   bot;
    ~*adsbot-google                bot;
    ~*googleother                  bot;
    ~*google-cloudvertexbot        bot;
    ~*feedfetcher-google           bot;
    ~*curl/                        script;
    ~*wget/                        script;
    ~*python-requests              script;
    ~*python-httpx                 script;
    ~*aiohttp                      script;
    ~*go-http-client               script;
    ~*node-fetch                   script;
    ~*axios/                       script;
    ~*undici                       script;
    ~*okhttp                       script;
    ~*java/                        script;
    ~*scrapy                       script;
    ~*headlesschrome               script;
    "~*bot\b|crawler|spider"      bot;
    "~*mozilla/.*(applewebkit|gecko|trident)"  browser;
}

# A present Signature-Agent header is a claim to verify, not proof.
map $http_signature_agent $bot_signed {
    default 1;
    ""      0;
}

log_format botlog '$remote_addr [$time_local] "$request" $status '
                  '"$http_user_agent" class=$bot_class signed=$bot_signed';
access_log /var/log/nginx/bots.log botlog;

Verifying a Claimed Bot

Classification tells you what a request says it is. This Python helper checks whether that is true, using reverse DNS with forward confirmation or the operator’s published IP ranges. Cache the ranges and refresh them daily rather than fetching them on every request. Feed the boolean result into classify_and_verify so a failed check is recorded as spoofed.

# verify.py: confirm a claimed crawler by reverse DNS + forward confirm, or by published ranges.
import ipaddress
import json
import socket
import urllib.request

RDNS_SUFFIXES = {
    "Googlebot": (".googlebot.com", ".google.com", ".googleusercontent.com"),
    "Bingbot": (".search.msn.com",),
    "Applebot": (".applebot.apple.com",),
    "CCBot": (".crawl.commoncrawl.org",),
    "YandexBot": (".yandex.ru", ".yandex.net", ".yandex.com"),
}

RANGE_URLS = {
    "GPTBot": "https://openai.com/gptbot.json",
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
    "ClaudeBot": "https://claude.com/crawling/bots.json",
    "Googlebot": "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
    "Bingbot": "https://www.bing.com/toolbox/bingbot.json",
}


def verify_rdns(ip: str, suffixes: tuple) -> bool:
    try:
        host = socket.gethostbyaddr(ip)[0].lower().rstrip(".")
        if not host.endswith(suffixes):
            return False
        # Forward-confirm: the hostname must resolve back to the same IP.
        infos = socket.getaddrinfo(host, None)
        return ip in {i[4][0] for i in infos}
    except (socket.herror, socket.gaierror):
        return False


def load_ranges(url: str) -> list:
    req = urllib.request.Request(url, headers={"User-Agent": "range-fetcher/1.0"})
    with urllib.request.urlopen(req, timeout=10) as resp:
        data = json.load(resp)
    nets = []
    for p in data.get("prefixes", []):
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            nets.append(ipaddress.ip_network(cidr))
    return nets  # cache these; refresh daily, not per request


def ip_in_ranges(ip: str, nets: list) -> bool:
    addr = ipaddress.ip_address(ip)
    return any(addr in n for n in nets if n.version == addr.version)

Sending Bot Hits to a Separate GA4 Property

If your marketing team wants bot trends in GA4, send them from the server with the Measurement Protocol to a separate property. Register bot_class and bot_name as event-scoped custom dimensions. Sample it or aggregate it if your volume is high, and keep the API secret out of the browser.

// ga4-mp.js (Node 18+): send a server-side classification to a SEPARATE GA4 property.
// Keep the API secret on the server. Never ship it to the browser.
const MEASUREMENT_ID = process.env.GA4_BOT_MEASUREMENT_ID; // e.g. G-XXXXXXXXXX
const API_SECRET = process.env.GA4_BOT_API_SECRET;
const ENDPOINT = process.env.GA4_MP_DEBUG ? 'debug/mp/collect' : 'mp/collect';

async function sendBotHit({ class: botClass, name }, pageUrl) {
  if (botClass === 'browser') return; // humans are measured by your normal tag
  const body = {
    // Bots have no _ga cookie, so use a stable pseudo client per bot name.
    client_id: `bot.${name.toLowerCase().replace(/[^a-z0-9]+/g, '-')}`,
    events: [{
      name: 'bot_request',
      params: {
        bot_class: botClass,          // register as an event-scoped custom dimension
        bot_name: name.slice(0, 100), // standard properties cap values at 100 characters
        page_location: pageUrl.slice(0, 1000),
        engagement_time_msec: 1,
      },
    }],
  };
  const url = `https://www.google-analytics.com/${ENDPOINT}?measurement_id=${MEASUREMENT_ID}&api_secret=${API_SECRET}`;
  const res = await fetch(url, { method: 'POST', body: JSON.stringify(body) });
  return ENDPOINT.startsWith('debug') ? res.json() : res.status;
}

module.exports = { sendBotHit };

How I Tested These Samples

I ran every sample before publishing. A shared test file of real and synthetic user agents covers every class, including the person-triggered fetchers that share a prefix with crawlers, a signed Chrome request, empty and unknown user agents, and generic bots. The PHP, Node, Python, and Go versions all return the same class and name for every case, and the Express middleware was tested against a live server. The nginx map was loaded in nginx 1.26 and checked with requests for each sample. The reverse DNS helper confirmed real Googlebot, Bingbot, and Applebot addresses and rejected a public resolver, and the Measurement Protocol payload validated cleanly against GA4’s debug endpoint. I also tested the verification step: a GPTBot user agent from an IP in the published ranges stays ai_training and verified, the same user agent with a failed range check becomes spoofed, a signed Chrome request becomes spoofed when the signature check fails, and an ordinary browser is not marked spoofed just because it was not on a crawler list.

One note on the generic pattern: matching bot, crawler, or spider can catch the occasional legitimate device or app name. Review what lands in that bucket before you block it.

Frequently Asked Questions

Can GA4 Show Me My Bot Traffic?

Not directly. GA4 automatically excludes known bots and does not let you see or disable that exclusion. Most crawlers never run your tag anyway, and GA4 has no user agent dimension. You need server, CDN, or middleware logs to see bot traffic, and you can optionally send summarized bot events to a separate GA4 property with the Measurement Protocol.

Does Blocking GPTBot Keep My Content Out of ChatGPT?

Only partly. GPTBot controls training use. OAI-SearchBot controls whether you appear in ChatGPT search. ChatGPT-User fetches pages when a person asks, and OpenAI says robots.txt rules may not apply to those requests. Decide on each one separately.

Should I Block AI Crawlers?

It depends on your business. Training crawlers take content and rarely send visitors, so many publishers block them. AI search crawlers and assistant fetchers can put you in front of buyers inside AI answers, so most SaaS vendors should allow them and rate limit expensive routes.

How Do I Know a Request Claiming to Be Googlebot Is Real?

Do a reverse DNS lookup on the IP, confirm the hostname ends in googlebot.com, google.com, or googleusercontent.com, then resolve that hostname and confirm it returns the same IP. Or check the IP against the IP ranges Google publishes. Never trust the user agent alone.

What Is the Signature-Agent Header?

It is part of Web Bot Auth, an IETF draft built on HTTP Message Signatures (RFC 9421). A bot that sends it is saying its request is signed and naming where its public keys live. It is only proof once you verify the signature against a key directory you trust.

Will Blocking Google-Extended Hurt My Search Rankings?

No. Google-Extended is a robots.txt control token, not a crawler. Google says it controls whether content crawled by Googlebot is used for Gemini training and grounding, and that it does not affect inclusion or ranking in Google Search.

What Is a Malicious Bot, and How Do They Hide?

A malicious bot is automated traffic meant to steal, defraud, or disrupt, including credential stuffing, scraping, inventory hoarding, card testing, ad fraud, form spam, vulnerability scanning, DDoS, and fake account creation. Many spoof a legitimate user agent such as Googlebot or ChatGPT-User, or they send an ordinary Chrome string. Trust a claimed identity only after you verify it, and use rate and behavior signals for requests that look like browsers.

Should I Just Block Every Bot or Every AI Crawler?

No. A blanket block hides you from AI search and assistants, breaks link previews, and can stop agents that are acting for real buyers. It also misses malicious clients that already look like people. Classify by purpose, verify the claim, and decide on assistants, AI search, and training crawlers separately.

Is It Enough to Allow Only Googlebot and Bingbot?

No. That policy keeps classic search but shuts out AI search crawlers and assistant fetchers that now drive discovery. It also does nothing about malicious bots that spoof ordinary browsers, and an allowlist of famous user agents is easy to forge unless you verify the request.

Related Articles