Alerts are only as useful as the rules behind them. An alert system that fires constantly becomes noise; one that fires too rarely misses incidents that compound while you’re looking the other way.
Before designing a strategy, settle what kind of alerting you have. In LLM Metrix, alerts are findings derived from a completed scan, not threshold rules you author. You choose which of the six alert types reach you, on which channels, at what magnitude, and which engines; most of the “how much counts as a change” numbers are now thresholds you set in Settings → Notifications. There is no rule builder and no per-prompt alert scoping; there IS a per-engine delivery filter and a pause (“snooze”) for the per-scan channels.
That shapes everything below, so this guide is in two halves: what the product raises for you, and what you still have to watch yourself.
What actually fires
Every completed scan is evaluated against a fixed rule set. Some rules read the scan on its own; five of the six toggles compare it against your previous scan, which means they say nothing at all on a project’s first run.
From the scan by itself
Zero-mention engine. An engine that mentioned your brand in none of the scan’s answers. This is the highest-value alert in the set, because it is exactly the failure that looks like nothing: no error, no empty page, just an engine quietly returning answers you are not in. It is also how a misconfigured or retired engine model presents, so read it as “investigate”, not automatically as “we lost ground”. When a baseline exists it sharpens its own wording to stopped mentioning you, which is a much stronger claim than the first-scan version and is worth reading literally.
Negative sentiment on an engine. An engine whose mentions of you skew negative across the scan.
A possible inaccurate claim. An answer carrying a medium or high accuracy risk about your brand, named with the engine and the claim. This is the brand-safety alert, and it is per-answer rather than per-scan; one scan can raise several.
A competitor out-mentioning you. The most-mentioned competitor in the scan appeared more often than you did across tracked queries. Informational rather than urgent: it is a standing state, not an event, so it keeps firing while it remains true.
A strong overall score. 70 or better raises a success-level alert. Worth knowing so you do not read every alert as bad news.
Score below your floor. If you set a score floor in Settings → Notifications, any scan whose score sits under it raises a warning: the one alert that is a level rather than a movement. Unlike the score-change rule it needs no previous scan to compare against, and it keeps firing on every scan while the score stays under the floor. It defaults off.
By comparison with your previous scan
These are the six you can switch on and off. The magnitudes are printed next to each toggle in Settings → Notifications and most are thresholds you set. The defaults are shown here:
| Alert | Fires when |
|---|---|
| Visibility score change | Your score moves 5 points or more (editable, 1–50) in either direction, or drops below your score floor. A fall is a warning, a rise a success |
| Citation change | A domain is cited that your previous scan did not cite, or one you were cited by stops being cited. Your own domain is called out separately from a third-party source |
| Competitor spike | A tracked competitor gains 10 or more percentage points (editable, 1–100) of share of voice |
| Sentiment shift | Engines describe you measurably less positively than last time (0.15 editable on the 0–1 scale), reported as a count of negative answers rather than an index |
| Position change | Your average position moves by a position or more (editable, 1–50), or your first-mention share drops by 10 or more points (editable, 1–100), between scans. A fall is a warning, an improvement a success |
| Hallucination detected | Governs the accuracy-risk and negative-sentiment findings above |
Three properties of that half are worth internalising before you build a process on it:
- A missing input produces silence, never a zero. No previous scan, a previous scan that delivered no answers, or a history row too thin to carry citations; in every case the affected rule emits nothing. The alternative, reading “we cannot see last week’s citations” as “last week had none”, would announce every domain in the current scan as newly cited on the first run after an outage.
- Turning an alert off changes delivery, not derivation. /dashboard/alerts shows every finding regardless of your notification settings, so a preference can never hide something from the dashboard.
- Competitor benchmark scans notify nobody, on any channel, by design. A benchmark measures someone else’s visibility; its findings are claims about them, not news about you.
What you have to watch yourself
Even with the cross-scan rules on, there is a gap between that list and a complete alerting strategy. Naming it is more useful than pretending otherwise, because each of these fails silently: nothing arrives, and nothing tells you nothing arrived.
Position drift confined to a few prompts. The position-change rule has a per-row form: a (query, engine) row that crosses a position band between scans (first → prominent, mid-list → fine-print, named → absent) is reported as a position-class-transition alert. What stays a read is drift within a band (second to third place, both prominent): that still needs the per-prompt rows on the Rankings page and the position trend chart, and it is the drift most likely to hide inside a flat mention rate.
A citation lost below the domain set. The citation rules diff the domain SET, so a domain that is still cited somewhere on the scan (just no longer for the prompt you care about) does not fire. Which page of yours an engine stopped citing for which question is still something you read from the Citations page or GET /api/v1/scans.
Anything scoped narrower than the project, other than engines. Delivery can be filtered per engine (Settings → Notifications → “Engines to alert about”), and the engine-shaped findings (an engine going silent, an engine-level sentiment or accuracy flag) are per engine by construction. Everything else is computed over the whole scan. A problem confined to three prompts or one market either moves the project-level number enough to fire or it does not, and mostly it does not.
Slow erosion under the threshold. Five points is a real move; four points a scan, four scans running, is a sixteen-point slide that never fires once. The trend charts are the only thing that catches a decline that never takes a big enough single step.
The practical shape of a strategy is therefore: let alerts catch the events, and put a recurring calendar item against the trends. Paid plans refresh daily and free weekly, so a fifteen-minute read after each scan lands covers all four.
Routing what does fire
Separate instant from digest. This is the one genuine configuration lever, and it is worth using deliberately. Alerting supports email and a generic outbound webhook, with per-scan email governed by a digest frequency of immediate, daily, weekly, monthly or off. Set it to immediate if you want brand-safety findings the day they appear; set it to daily, weekly or monthly if you want the trend rather than the events. A warning-level finding reaches you by email regardless of that setting, which is what keeps an accuracy risk from sitting in a monthly batch.
Email for people, webhook for the system. Email is where a finding gets a human owner within the hour. For a team, it is also the place where whoever is on it sees it. The webhook is what puts alerts into your own tracking system, which is also the honest workaround for everything in the previous section: if you want threshold rules, receive scan results at your own endpoint and apply your thresholds there.
Notification settings are per user, not per market or project. A five-person team’s members each configure their own; there is no way to route one project’s alerts to one person and another’s elsewhere. If you need that split, the webhook is the seam.
Alert triage workflow
When an alert fires, the goal is to answer three questions within 24 hours:
1. Is this real or transient?
AI engines produce variable outputs, and that variance is a property of the model rather than something you can tune away from outside it. Gemini’s text generation documentation describes temperature, topP and topK as generation-time settings, and notes they are deprecated and ignored on its newest models. Each prompt is asked once per engine per scan, so a single scan is a single draw. Re-run the query yourself a few times, or run an on-demand scan, before treating one reading as a shift.
2. What specifically changed? Compare the current answer against what you recorded previously; there is no stored response diff, so this depends on your own notes or an export. A different competitor? A different source cited? New phrasing in your brand description?
3. Who owns the remediation? Position drift → content team. Brand safety → brand/comms + content team. Competitor gain → content strategy + SEO. Score drop → investigate root cause before assigning.
Document this triage in your tracking system before closing the alert. Patterns across multiple alerts (the same competitor gaining repeatedly, the same engine consistently returning nothing) are worth more than any single alert.
What not to chase
Minor sentiment fluctuations. Sentiment varies naturally as answers vary. The negative-sentiment alert fires on an engine’s aggregate for a scan, which is already coarser than a single response, but one scan is still one reading. Act on it when it repeats.
Normal competitor co-presence. A competitor appearing in an answer you also appear in is not an emergency. The competitor alert deliberately fires on being out-mentioned overall, not on co-presence.
A single scan’s outlier on a low-priority prompt. Non-determinism produces outliers. Don’t let one unusual answer to a question you don’t prioritise consume attention you owe a real event.
Frequently Asked Questions
Can I configure alert thresholds in LLM Metrix?
Mostly, yes. You choose which of the six alert types reach you and on which channels, and you set how much counts as a change: the visibility-score move (1–50 points), the competitor share-of-voice gain (1–100 points), the sentiment shift (0.05–1 on the 0–1 scale), the position move (1–50 positions), the first-mention drop (1–100 points) and an absolute score floor (0–100; 0 = off). There is still no rule builder and no per-prompt scoping, but the per-engine delivery filter and the pause control cover the two common noise complaints. If you genuinely need your own logic, route alerts to your own endpoint via the outbound webhook and apply it there.
What AI visibility events are worth watching?
Four categories: brand safety (factually wrong claims about your brand), score erosion, competitor gains, and position drift (your mention tier dropping). All four are raised automatically: the accuracy-risk alert, the score-change rule, the competitor share spike and the position-change rule (average rank, first-mention share, or a per-query band slide). A citation you lose is now flagged too. What still needs your eyes: erosion too gradual to cross the threshold in a single scan, and drift that stays within one position band.
How do I know if an alert reflects a real change or just AI variability?
Each prompt is asked once per engine per scan, so any single alert rests on one reading. Re-run the query a few times by hand, or run an on-demand scan, and see whether the finding repeats. Confirming before acting is the core of effective triage, and it is why the trend across several scans, rather than any one of them, is the thing to steer by.
Do competitor benchmark scans send alerts?
No, on any channel, by design. A benchmark scan measures a competitor’s visibility with them as the subject, so the findings it records are claims about them rather than news about you. Check those results in the dashboard when you run one; nothing will be pushed to you.