Traffic · Crawlers & referrals

Which AI reads your site,
and which sends people.

Two halves of the same question. AI crawlers hit your pages to train models, build answer indexes or answer someone's live question; AI answer surfaces then send visitors back. This surface measures both, and the ratio between them.

Start for free Read the guide

Free forever plan — no card required

app.llmmetrix.com/dashboard/traffic

Illustrative

AI crawlers · 30 days

GPTBot

OpenAI · training

Verified4,182

OAI-SearchBot

OpenAI · search index

Verified1,905

PerplexityBot

Perplexity · search index

Verified1,240

ClaudeBot

Anthropic · training

Unverified864

Referrals by source

ChatGPT

46%

Perplexity

28%

Gemini

15%

Claude

11%

Why it matters

Built to move your AI visibility from invisible to inevitable.

A crawler taxonomy, not a user-agent dump

Crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, ByteDance, Meta and Common Crawl, each classified by operator and by why it visits: model training, answer-index building, or a real-time fetch for a live question.

Verification, because a user agent is free text

Anyone can claim to be GPTBot. Requests are checked against each operator's published IP-range file, or reverse-resolved and forward-confirmed the way Googlebot verification works. Operators publishing no mechanism stay honestly marked unverified.

Referral attribution that survives stripped referrers

ChatGPT, Perplexity, Gemini, Claude, Microsoft Copilot and Meta AI are matched by referrer host or by a UTM tag you seeded — and the explicit tag wins, because AI apps routinely strip the referrer entirely. A link builder generates the tagged URLs.

Daily rollups and a crawl-to-refer ratio

Crawls, unique paths and operator-attributed referrals per bot per day. A ratio over zero referrals reads as "no referrals yet", never as a plausible-looking zero — and an operator's referrals are counted once per day, not once per bot.

Crawled but never referred

The pages AI crawlers read that no AI referral ever landed on — the content the engines consume without sending anyone back. That list is the optimisation backlog.

Two ingestion paths, each fenced

A first-party edge beacon for referrals, and server access-log upload for crawler hits. The beacon key is write-only and bound to one project; log upload additionally requires proven domain ownership.

Who it's for

A daily edge for the teams shaping AI answers.

Technical SEOs

See exactly which AI crawlers reach the site, how often, and on which paths — before deciding what to allow, block or restructure.

Growth teams

Attribute real sessions to ChatGPT, Perplexity, Gemini, Claude, Copilot and Meta AI, including the ones that arrive with no referrer at all.

Content owners

Find the pages AI reads heavily and refers from never, and treat that gap as a content brief rather than a mystery.

FAQ

Questions, answered.

How do you tell a real AI crawler from something pretending to be one?

A user-agent string is free text, so it is treated as a claim rather than a fact. Where an operator publishes a machine-readable IP-range file, the request IP is tested against it. Where an operator documents reverse DNS instead, the IP is reverse-resolved, required to land under a documented hostname suffix, and then forward-confirmed back to the same IP — a spoofer controls their own reverse DNS record, so the lookup alone proves nothing. Verification never blocks ingestion: any failure records the hit as unverified.

Some bots always show as unverified. Is that a bug?

No. Several operators publish no verification mechanism at all, so a user-agent match from them can never be confirmed and is recorded as unverified rather than quietly upgraded. Saying so is the honest answer; claiming verification we cannot perform would be worse than saying nothing.

Why does uploading server logs require domain verification?

Because ingesting access logs for a domain you have not proven you control is a data-poisoning primitive — those rows become that project's crawler analytics. Scanning a domain is observation of public data and is deliberately ungated, but log upload requires at least the meta-tag tier of ownership proof: enough to show control, cheap enough to do in five minutes without a DNS ticket.

The beacon key sits in my page source. Is that safe?

It is public by construction, so it is built to be worth nothing to an attacker. It is a write-only collector key: the read API rejects it outright, it can only append referral rows to the single project it is bound to, the target project is derived from the key itself rather than from the request body, and a browser Origin check rejects it when used from a site that is not yours.

Which plans include this?

The attribution tag is available on paid plans, and server-log ingestion — the heaviest path — is an Agency capability. The gate is enforced at ingestion rather than at the dashboard, so a workspace that downgrades stops accruing new rows while its existing history stays readable.

Keep exploring

Related features

See all features

Citation Intelligence

Discover which pages and sources AI engines are citing in answers about your space — and what to do about it.

Search-AI Gap

Join your Google Search Console queries to your scanned prompts and see the two places the surfaces disagree — ranked but not cited, and cited but not ranking.

Search Performance

Google Search Console clicks, impressions and impression-weighted position, with query coverage stated rather than hidden. Available on every plan.

What's changed— recent releases that touch this feature

v0.0.6

AI Traffic Analytics & Connectors

v0.0.9

Measured Status, Readable Charts & Polite Crawling

v0.0.10

Schema Validator, Ninth Free Tool

v0.0.1

Launch

Key concepts— terminology used in this feature

Content Audit Crawl Budget Crawlability GPTBot Internal Linking llms.txt Open Graph PerplexityBot

In depth— analysis and real-world examples

Should You Block AI Crawlers? The Trade-Off Nobody Quantifies

Blocking GPTBot is not one decision — it's three, because training crawlers, search crawlers and user-triggered fetchers are different bots with different costs. Conflating them is how publishers accidentally removed themselves from AI answers.

8 min read

What SEO Teams Should Stop Doing First

Most AEO advice for SEO professionals is additive — here are more tactics. The subtractive list is more useful: which well-drilled habits carry over to answer engines, which do nothing, and which actively mislead. Covers rank tracking, keyword-led prioritisation, E-E-A-T audits, crawler access and schema.

8 min read

Your Server Logs Know Which AI Crawlers Visited. Most Teams Never Look.

Access logs are the only first-party evidence you have that AI systems are reading your site at all. Here's what they can prove, the verification step almost everyone skips, and the two questions logs genuinely cannot answer.

8 min read

Start on the free plan. No card, no time limit.

The AI era of search
is already here

Every day without LLM Metrix is a day your competitors are studying the same AI answers you haven't seen yet. Start free today — no card required.

Get started free Book a demo

Free forever plan — no card required

Traffic · Crawlers & referrals

Which AI reads your site,
and which sends people.

Start for free Read the guide

Free forever plan — no card required

app.llmmetrix.com/dashboard/traffic

Illustrative

AI crawlers · 30 days

GPTBot

OpenAI · training

Verified4,182

OAI-SearchBot

OpenAI · search index

Verified1,905

PerplexityBot

Perplexity · search index

Verified1,240

ClaudeBot

Anthropic · training

Unverified864

Referrals by source

ChatGPT

46%

Perplexity

28%

Gemini

15%

Claude

11%

Why it matters

Built to move your AI visibility from invisible to inevitable.

A crawler taxonomy, not a user-agent dump

Verification, because a user agent is free text