If you’ve run the same prompt twice and gotten different AI responses — different brand mentions, different citation order, different recommendations — you’re not imagining things. AI engines are inherently non-deterministic. Understanding why — and how AI search works under the hood — is important for interpreting your monitoring data correctly and not chasing false signals.
The core reason: temperature
LLMs generate text token by token. At each step, the model calculates a probability distribution over all possible next tokens, then samples from that distribution. Temperature is the parameter that controls how that sampling works:
- At low temperature, the model almost always picks the highest-probability token — responses are predictable and consistent
- At high temperature, the model sometimes picks lower-probability tokens — responses are more varied and creative
Most AI search engines operate at low-to-medium temperatures for factual queries, but not at zero. Even at temperature 0.2, minor variations accumulate across hundreds of tokens, producing meaningfully different outputs across runs.
This is documented behaviour, not an implementation quirk. Google’s text generation documentation describes temperature, top_p and top_k as the parameters governing exactly this sampling, and — notably — now advises leaving them at their defaults for its newest models, warning that lowering temperature can degrade output rather than stabilise it. Determinism is not simply available on request.
What this means for your brand: A brand that appears in 8 out of 10 readings of a query has a strong association for that query. A brand that appears in 3 out of 10 has a weaker, more marginal association — it falls within the model’s consideration set but isn’t the obvious choice. Variability across readings tells you how firmly established your brand is for each query.
One thing to be clear about before you plan around that: a scan asks each prompt exactly once per engine. There is no repeated sampling inside a single run, so the ten readings above are ten successive scans, not ten runs of one scan. That is a deliberate cost decision — repeated sampling would multiply the credits a prompt consumes by the sample size — and it means the sample you build up accumulates over weeks rather than arriving all at once.
Retrieval variability (RAG engines)
For engines that retrieve live web content, the retrieval step introduces its own variability:
Index freshness: The retrieval index is updated continuously. A page that was just crawled may appear in retrieval results that wouldn’t have included it 6 hours ago. This is generally positive — it means fresh content is reflected quickly — but it means the source set for any given query can shift between runs.
Tie-breaking in ranking: When two chunks have similar relevance scores, small floating-point differences in the re-ranking computation can flip their order. The chunk that lands at position 5 vs. position 6 may or may not make it into the final context window.
Retrieved content changes: If the actual pages in the index were updated between runs — a competitor published a new article, your own page was refreshed — the content that gets retrieved changes, which can change the generated response.
Model updates
AI providers update their models frequently — sometimes with public announcements (“GPT-4o update”), sometimes silently. A model update can change:
- How the model weighs different brand associations
- Which sources it prefers in retrieval
- How it frames recommendations in your category
- Its safety and neutrality policies around product recommendations
A sudden unexplained shift in your monitoring data — particularly if it happens across many queries simultaneously — is often a model update rather than anything you did. Spotting one is a manual read: compare your visibility-score and share-of-voice trends against the dates of known model releases. LLM Metrix does not detect these shifts for you — nothing in the product flags a trend as anomalous, so a large simultaneous move is something you have to notice yourself.
Query phrasing sensitivity
The exact phrasing of a query affects which brands the model surfaces, even for semantically equivalent prompts. “Best project management software” and “top tools for managing projects” may retrieve different content chunks and activate different model associations — producing different brand mention patterns.
This is why monitoring should cover a cluster of semantically related prompts for each topic area, not a single exact-match query. The aggregate view across query variants gives a more stable signal than any single phrasing.
How to interpret variable monitoring data
Don’t react to single-scan results. One scan showing your brand missing is not a crisis; one scan showing a competitor in first place is not confirmation of a trend. Patterns across many scans are the signal.
Look at trends, not snapshots. LLM Metrix re-runs your whole tracked prompt set on your plan’s scan cadence — weekly on every paid plan — and charts your visibility score and share of voice across scans over time. Both are project-level series, which is what makes them stable: they aggregate every prompt and engine in the scan, so one prompt’s run-to-run wobble barely moves them.
Per-prompt position is a latest-scan reading. The Rankings page shows where you placed for each prompt and engine in the most recent scan; there is no per-prompt chart across scans. To compare a single prompt’s position over time, read the scan history through GET /api/v1/scans — it returns per-answer rank across up to 100 past scans — or ask the assistant for the change since the previous scan. The product does not average a prompt across scans for you.
Treat trend direction as the meaningful signal. If your mention rate moves from 45% to 38% over four scans, that’s a real trend — sustained, directional movement across many readings. A one-scan dip from 45% to 42% may just be noise.
Investigate sudden cliff-edges. If your metrics drop sharply between two consecutive scans rather than gradually — especially across prompts that have nothing to do with each other — that’s likely a model update or retrieval index change rather than a competitive or content-driven shift. Check whether other brands in your category saw similar movements (the competitor benchmarking view helps here).
The right baseline expectation
For most brands monitoring AI visibility (on a weekly cadence, so one reading per week):
- ±5 percentage points scan-over-scan variability in mention rate is normal noise
- ±1 position tier variability in mention position between scans is expected
- Consistent directional movement over 4+ scans is a real trend worth acting on
- Shifts across unrelated prompts (everything moving together at once) typically indicate a model event, not a content event
Frequently Asked Questions
Is it a problem that the same query gives different answers each time?
No — this is expected behavior, not a bug. LLMs sample tokens from a probability distribution, so even at low temperature minor variations accumulate across a response. The right approach is to track each query across repeated scans and read the aggregate pattern rather than reacting to any single reading.
How many times should I run a query before trusting the result?
Each tracked query is asked once per engine per scan, so your sample grows one reading per scan rather than through repeated runs. Read the pattern across successive scans: a brand named in 8 of the last 10 scans has a strong association for that query, while one named in 3 signals a weak, marginal one. You do that read yourself — from the scan history in GET /api/v1/scans or by comparing scans in the dashboard — since LLM Metrix charts the score and share-of-voice trends at project level and does not average an individual query across scans.
How do I tell normal variation from a real change?
Treat sustained directional movement over four or more weeks as a real trend, while week-over-week swings within roughly ±5 percentage points of impression rate are normal noise. A sharp same-week drop across unrelated query clusters usually points to a model update or retrieval-index change rather than a content or competitive shift.
Why does rephrasing a query change which brands appear?
Different phrasings retrieve different content chunks and activate different model associations, even when prompts are semantically equivalent. That’s why monitoring should cover a cluster of related prompts per topic rather than a single exact-match query — the aggregate view across variants gives a far more stable signal than any one wording.