# Filtering Bot Traffic > Identify, measure, and exclude bot / crawler / AI-agent traffic in PostHog web and product analytics using the traffic classification surface (the isLikelyBot / getTrafficType HogQL functions and the $virt_* virtual properties). Use when the user asks to "exclude bots", "filter out crawlers", "remove bot traffic from my numbers", "how much of my traffic is bots / AI crawlers", "is GPTBot / ChatGPT / Claude hitting my site", "break down traffic by human vs bot", or wants clean human-only counts in an insight or dashboard. For the real-time Live tab bot tiles, use exploring-live-traffic instead. - Skill: `posthog/filtering-bot-traffic` (Agent Skill) - Install (CLI): `npx skillmds add posthog/filtering-bot-traffic` - Raw SKILL.md: https://api.skillmd.com/api/skills/posthog/filtering-bot-traffic/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Data & Analytics - Author: posthog (https://skillmd.com/u/posthog) - Updated: 2026-09-09 - Page: https://skillmd.com/skills/posthog/filtering-bot-traffic --- # Filtering and measuring bot traffic PostHog classifies every request by user agent so you can tell humans apart from bots, crawlers, and AI agents anywhere HogQL runs — the SQL editor, insights, trends, and Web analytics breakdowns. This skill teaches you (the agent) how to use that classification to: - exclude bots so analytics reflect human traffic only - measure how much traffic is automated, and which bots / operators are responsible - separate AI-agent traffic (worth measuring) from noise (worth dropping) - pick the right surface — virtual properties for the insight builder, functions for raw SQL For real-time ("right now", last 30 min) bot questions and the Live tab tiles, use the **exploring-live-traffic** skill instead. This skill is for historical windows, saved insights, dashboards, and filtering. ## When to use this skill Use it when the user wants to: - exclude or filter out bots ("remove bots from my pageviews", "humans only") - quantify automated traffic ("what % of traffic is bots?", "how much is AI crawlers?") - find which bots hit them ("which crawlers visit us?", "is ChatGPT reading our docs?") - break a trend down by traffic type or bot name - measure AI-agent / AI-search traffic specifically (AEO / answer-engine visibility) Do **not** use it for the Live tab, real-time numbers, or the per-minute bot charts — that is exploring-live-traffic. ## The classification surface Two equivalent ways to reach the same classification. Prefer **virtual properties** in the insight builder and filters; use **functions** in hand-written SQL or when you need a value the virtual properties don't expose. ### Virtual properties (insight builder, filters, breakdowns) These read the user agent for you (falling back from `$raw_user_agent` to `$user_agent`), so you don't pass anything in. Available wherever you pick an event property. | Property | Value | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `$virt_is_bot` | boolean — `true` for bots / crawlers / automation | | `$virt_traffic_type` | `Regular`, `AI Agent`, `Bot`, or `Automation` | | `$virt_traffic_category` | finer category, e.g. `ai_crawler`, `ai_search`, `ai_assistant`, `search_crawler`, `seo_crawler`, `social_crawler`, `monitoring`, `http_client`, `headless_browser`, `no_user_agent`, `regular` | | `$virt_bot_name` | display name, e.g. `Googlebot`, `GPTBot`, `ClaudeBot` | | `$virt_bot_operator` | company behind the bot, e.g. `Google`, `OpenAI`, `Anthropic` | ### HogQL functions (raw SQL) Pass the user agent explicitly. Use `coalesce(nullIf(properties.$raw_user_agent, ''), properties.$user_agent)` to cover both server-side (`$raw_user_agent`) and JS SDK (`$user_agent`) captures. The `nullIf` keeps an empty `$raw_user_agent` from shadowing a real `$user_agent` and being misread as a bot — this mirrors the expression the virtual properties use internally. | Function | Returns | | ------------------------ | ---------------------------------------------------------------------------- | | `isLikelyBot(ua)` | `true` if the UA matches a bot/automation pattern (empty UA counts as a bot) | | `getTrafficType(ua)` | `AI Agent` / `Bot` / `Automation` / `Regular` | | `getTrafficCategory(ua)` | subcategory; `regular` for humans | | `getBotType(ua)` | same subcategory but empty string for humans — handy for filtering | | `getBotName(ua)` | bot name; empty for humans | | `getBotOperator(ua)` | operator/company; empty for humans | ## Traffic types — what to keep vs drop `getTrafficType` / `$virt_traffic_type` sorts every request into four buckets. The default move differs per bucket — don't treat them all as noise: | Type | What it is | Default move | | ------------ | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | `Regular` | Human visitors | Keep | | `AI Agent` | AI crawlers, AI search, AI assistants (GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User) | Often **measure**, don't drop — these are how AI tools find and cite content | | `Bot` | Search crawlers, SEO tools, social previews, monitoring (Googlebot, AhrefsBot, Pingdom) | Exclude from human metrics; track separately for SEO | | `Automation` | HTTP clients and headless browsers (curl, python-requests, Puppeteer) | Usually noise — exclude | ## Recipes ### Exclude bots from an insight (humans only) Add a property filter `$virt_is_bot` `exact` `false`: ```json { "key": "$virt_is_bot", "value": ["false"], "operator": "exact", "type": "event" } ``` Drop it into any TrendsQuery / FunnelsQuery / etc. `properties`. Visitor, session, and pageview counts then reflect human traffic only, without changing stored data. To exclude a narrower slice (e.g. keep AI agents but drop monitoring + automation), filter on `$virt_traffic_type` or `$virt_traffic_category` with `operator: is_not` instead. ### What share of traffic is automated Break a pageview trend down by `$virt_traffic_type`: ```json { "kind": "TrendsQuery", "dateRange": { "date_from": "-30d" }, "series": [{ "kind": "EventsNode", "event": "$pageview", "math": "total" }], "breakdownFilter": { "breakdown": "$virt_traffic_type", "breakdown_type": "event" }, "trendsFilter": { "display": "ActionsBarValue" } } ``` ### Which bots / operators are hitting us Filter to bots and break down by name (or `$virt_bot_operator` for company-level): ```json { "kind": "TrendsQuery", "dateRange": { "date_from": "-30d" }, "series": [{ "kind": "EventsNode", "event": "$pageview", "math": "total" }], "properties": [{ "key": "$virt_is_bot", "value": ["true"], "operator": "exact", "type": "event" }], "breakdownFilter": { "breakdown": "$virt_bot_name", "breakdown_type": "event", "breakdown_limit": 25 }, "trendsFilter": { "display": "ActionsBarValue" } } ``` ### Measure AI-agent traffic specifically Filter `$virt_traffic_type` `exact` `AI Agent`, break down by `$virt_bot_operator` to see which tools (OpenAI, Anthropic, Perplexity, …) read your site and which pages they hit. ### Raw SQL equivalents ```sql -- human pageviews only SELECT count() AS human_pageviews FROM events WHERE event = '$pageview' AND NOT isLikelyBot(coalesce(nullIf(properties.$raw_user_agent, ''), properties.$user_agent)) -- top bots by hits SELECT getBotName(coalesce(nullIf(properties.$raw_user_agent, ''), properties.$user_agent)) AS bot, getBotOperator(coalesce(nullIf(properties.$raw_user_agent, ''), properties.$user_agent)) AS operator, count() AS hits FROM events WHERE event = '$pageview' AND isLikelyBot(coalesce(nullIf(properties.$raw_user_agent, ''), properties.$user_agent)) GROUP BY bot, operator ORDER BY hits DESC ``` ## Seeing bots that don't run JavaScript Most crawlers and AI agents never execute JS, so `posthog-js` never fires a `$pageview` for them — they're invisible to client-side analytics. To measure them, the project must forward server access logs as `$http_log` events carrying `$raw_user_agent`. If a user asks "why don't I see GPTBot when I know it's crawling us?", the answer is almost always: no `$http_log` ingestion. Point them at server-side capture (the **Vercel logs** source, an edge worker, or the capture API) before building bot insights. ## Gotchas - **Needs a captured user agent.** Classification is computed at query time from the event's `$raw_user_agent` / `$user_agent`, so it works on any historical event — there's no need to restrict `dateRange.date_from`. The one requirement is that a user agent was captured; events from sources that never set one can't be classified (and empty UAs fall through to `Automation` / `no_user_agent`, below). - **`isLikelyBot` is "likely".** Detection is a user-agent heuristic — some bots spoof real browser UAs, and some legit tools use bot-like ones. Treat it as best-effort, not ground truth. - **Empty user agent = bot.** Requests with no UA (server-to-server, misconfigured SDKs) classify as `Automation` / `no_user_agent`, so `isLikelyBot` returns `true`. - **Don't silently drop the host filter.** If the user is scoped to one domain, inherit `$host` in `properties` — leaving it out changes the answer. - **Bot definitions evolve.** The detected-bot list changes over time, so re-running the same query later can classify older events differently.