Monitoring AI Traffic and Crawlers

Visibility inside AI answers is one half of the picture. The other half is what actually happens to your site: which assistants send real visitors, which crawlers read your pages, and how far apart those two numbers are. The AI Traffic page answers that, and it needs data you supply — nothing here is derived from scans.

There are two ingestion paths and they measure different things. A JavaScript tag records referrals (a human arriving from ChatGPT, Perplexity, Gemini, Claude, Copilot or Meta AI). Server access logs record crawls (a bot requesting your pages). You can run either alone, but the most useful number on the page — crawl-to-refer — needs both.

Step 1: Open AI Traffic and choose a window

AI Traffic sits in the Monitor group of the sidebar. Its subtitle names exactly what it attributes: visits from ChatGPT, Perplexity, Gemini, Claude, Copilot and Meta AI. Range buttons on the right offer 7d, 30d and 90d.

Until data lands you will see “No AI traffic recorded in the last 30 days” and a pointer to the setup sections further down the page — those stay visible whether or not you have data, so this is also where you come back to re-copy a snippet.

If your workspace is on the free plan you will also see a banner explaining that AI traffic attribution is a paid feature and that the collector is not accepting new events, with a link to upgrade. Existing history stays visible; only new collection stops.

Four stat cards summarise the window: AI Referrals, Top Source (with its share), Crawler Hits (with a bot count) and Crawl-to-Refer, expressed as crawls per referral.

Step 2: Install the attribution tag

Scroll to Data sources → Install the attribution tag.

Click Generate write key first. That mints a collector-kind API key scoped to the current project and drops it straight into the snippet above the button. The key is shown once — copy the snippet now.

The snippet is three lines:

<script async src="https://app.example.com/tag.js"
        data-project="YOUR-PROJECT-ID"
        data-key="llmx_YOUR-WRITE-KEY"></script>

Paste it into your site’s HTML, or into a GTM Custom HTML tag. Copy puts it on your clipboard.

Publishing a key in page source sounds alarming, and the design answer is worth understanding: a collector key is write-only. It is rejected by every read route in the public API, and the project it writes to is taken from the key’s own binding, never from the request body — so nobody can point a scraped key at another tenant’s project. That is also why you should not reuse an ordinary API key here; the collector split exists precisely so the credential in your page source cannot read your scans.

Step 3: Know what the beacon does and does not record

On each page load the tag reads document.referrer and any utm_* parameters and beacons them to the collector. It sets no cookies. It sends the path of the landing page — never the query string or fragment, which can carry personal data — plus the referrer host and the UTM fields.

Classification then decides whether the visit counts:

• UTM wins over the referrer. Several assistants and most mobile apps strip the referrer entirely, so an explicit utm_source you seeded is the more reliable signal, and it is trusted when the two disagree.
• Non-AI visits are acknowledged and deliberately not stored. This is not general web analytics; the referrals table holds AI-attributed visits only.
• The origin is checked. A browser cannot forge the Origin header, so a key embedded on an unrelated site is refused. Server-side senders that omit the header are still allowed.

Step 4: Add server logs for the crawler side

Crawler data cannot come from a browser tag — bots do not execute JavaScript — so it comes from your access logs, uploaded to POST /api/ingest/logs with an ordinary Authorization: Bearer llmx_… API key. There is no upload widget in the dashboard today; this is an API call you script or run from your log host.

Three gates apply, and the first surprises people:

1. Domain verification tier 1 or higher. Uploading logs is a claim that this traffic belongs to your site, so you must have proved control of the domain first — see Verifying Your Domain. Without it the endpoint returns domain_not_verified and tells you where to request a token.
2. The Agency plan. Log ingestion is the heaviest path in the product and is gated accordingly.
3. Size and rate. Up to 2,000,000 characters of log content per upload (4 MB of raw body), ten uploads per minute per key.

Set format to combined for NGINX/Apache combined logs, json for one JSON object per line (Vercel and CDN drains), or leave it as auto, which decides per line. Parsing happens in a background job, so the response is a jobId you can follow on the Activity page.

Uploads are idempotent: the job key is derived from the project plus a hash of the content, so re-sending the same file collapses onto the existing job instead of double-counting your crawlers.

Cloudflare zone analytics is a third source, connected through POST /api/integrations/cloudflare with a scoped Cloudflare token (Zone → Analytics: Read) and your 32-character zone id. It is gated to the same plan as log ingest, and there is no card for it in Settings today.

Step 5: Read the crawler table

Crawler frequency by bot lists what LLM Metrix classified from your logs, with columns for Bot, Crawls, Peak daily pages, Referrals and Crawl : Refer.

The taxonomy distinguishes crawlers by why they came, which changes what you do about them: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider and CCBot are training crawlers; OAI-SearchBot, PerplexityBot and Amazonbot build answer indexes; and ChatGPT-User, Claude-User and Perplexity-User are live fetches made because a real person just asked a question. That last group is the one worth optimising for first — it is a user waiting on an answer, not a corpus build.

A user-agent string is free text, so requests are verified where the operator makes it possible: published IP ranges, or reverse DNS with forward confirmation. OpenAI, for instance, publishes the IP ranges its crawlers use precisely so that site owners can tell a real GPTBot from something wearing its name. Verification never blocks ingestion — anything uncertain is simply recorded as unverified.

Which crawlers you allow is a separate decision from measuring them, and it is made in robots.txt. Every outbound fetch this product makes obeys RFC 9309, the robots.txt standard, and multi-engine monitoring shows the answer-side consequence of whatever you decide to allow.

Two panels turn this into work. Top AI-referred landing pages shows where AI answers send your visitors. Crawled but never referred shows pages AI reads without sending anyone back, which the panel calls your optimization backlog, and which is the most directly actionable list on the page.

Step 6: Close the loop with UTM-tagged links

Below the tag section, UTM-tagged links for AI seeding generates ready-to-paste URLs. Because assistants routinely strip the referrer, a tagged link is often the only durable attribution signal you will get. Use them anywhere an assistant might cite you — docs, directories, community answers, your own content — and the beacon will classify the arrival by its UTM even with no referrer.

Step 7: Where to go next

Read the crawl-to-refer ratio as a question rather than a score: a high number means AI reads a lot and returns little, which is a content and citation problem, not a traffic problem.

• Tracking AI referral traffic — the measurement problem this page exists to solve.
• The AI crawlers guide — who each bot is and what to allow.
• Do AI crawlers respect robots.txt? — before you decide to block one.
• Verifying Your Domain — the prerequisite for log upload.