New: Per-Scan Email Alerts for AI Visibility Changes

The Problem with Weekly Dashboards

LLM Metrix has always had a weekly digest email. Users told us it was useful, but also that they’d notice a major visibility drop on Friday, try to figure out what happened, and discover it had started on Monday. Five days of confusion baked in before anyone knew there was a problem.

For most visibility changes, a weekly cadence is fine. Scores don’t move dramatically day-to-day under normal conditions. But some changes matter most: an engine that stops mentioning you entirely, an engine that starts describing your brand negatively, an answer that asserts something about you that isn’t true. For those, waiting for the digest is too slow.

Note the precise thing being fixed, because it is narrower than “faster monitoring”. The weekly digest fires on Mondays, on its own schedule, regardless of when your scans actually ran. So a scan on Tuesday that found something sat on that finding for six days before anyone was told. Per-scan alerting decouples the two: the finding leaves when the scan lands. It does not make the scan happen sooner: that is your plan’s cadence, daily on every paid plan, plus whatever you run by hand.

The interesting part of that work turned out not to be the sending. It was deciding what not to send.

The failure mode we were actually designing against

An alerting system has two ways to fail, and only one of them is obvious.

The obvious failure is missing something. You wanted to know that an engine stopped citing you, and nobody told you for six days.

The non-obvious failure (the one that kills alerting systems in practice) is telling you everything. A channel that fires on every scan trains the recipient to filter it, and a filtered alert is functionally identical to an alert that was never sent, except that you are now paying for the illusion of coverage.

AI visibility monitoring inherits this problem in an unusually sharp form, because AI answers are non-deterministic. The same prompt asked twice can come back differently worded, with a different set of cited sources, without anything about your brand or the web having changed. A system that treated every wobble as news would fire constantly and mean nothing.

That constraint is still doing work in the alerts we have added since. The one that watches your citation set for domains that were not there last time is deliberately not warning-level, and the reason is exactly the sentence above: a new source is worth recording and worth reading on a Tuesday afternoon, but some real fraction of any citation-set change is sampling noise, so it has not earned an interruption. Severity is where that judgement lives.

So the design question is not “how fast can we notify?” It is “what is genuinely news about your brand?” Every rule below falls out of that one question.

What actually triggers a per-scan email

Two independent switches can fire a scan email, and either one is sufficient.

Email digest set to “After each scan.” This is the literal reading: a summary email on every completed scan. If your digest frequency is instead daily, weekly or monthly, per-scan email is off and you are batched into the digest cron, which runs the weekly send on Mondays and the monthly send on the first of the month. Setting frequency to “off” suppresses it entirely.

Alert emails. This one is cadence-independent. It means “email me when a scan surfaces something that needs attention,” and it fires only when the scan produced a warning-level alert. If you are on the weekly digest and a Tuesday scan finds a problem, you hear about it on Tuesday.

That second switch is the one worth understanding, because it is what makes a slow default cadence safe. You do not have to choose between a quiet inbox and finding out late.

What counts as warning-level

Alerts are derived from the scan itself and the one before it, and each carries a severity. You are not writing the rules: what you get to choose is which of the findings below are worth interrupting you for, with a switch per type in Settings → Notifications. The magnitude that counts as a change is ours, for the reason the rest of this post is about: a threshold you guessed at produces either a channel you learn to ignore or a silence you trust wrongly.

One warning has no switch and is not meant to: an engine that mentions you nowhere always reaches you. A surface going silent across your entire prompt set is the finding this product exists to catch, and it should not be possible to turn it off while tidying a noisy inbox.

The warning-level ones are the states that are actually bad. Three read the completing scan alone:

• An engine did not mention your brand for any tracked query in that scan (worded more sharply when it did in your previous scan), because “stopped” is a stronger claim than “doesn’t” and we only make it when we can back it.
• An engine describes your brand with negative sentiment.
• A brand-safety check flagged an answer carrying a medium or high accuracy risk: an engine asserting something about you that looks wrong.

Three more compare the two scans, and are silent until there is a previous one to compare against:

• Your visibility score fell by more than a set margin, quoting both figures.
• A competitor gained more than a set margin of share of voice.
• Engines describe your brand less positively than they did last time.

A strong overall result also produces an alert, but it is success-severity. So does a score that rose, and so does your own domain turning up in citations that did not carry it before. They appear on your alerts surface and they do not email you. Good news is not urgent, and a system that emails you when things are fine is a system you stop reading. Brand-safety monitoring is the surface those warnings feed, and the alert strategy guide goes deeper on deciding which changes deserve to interrupt you at all.

What never notifies anyone

Competitor benchmark scans notify nobody, on any channel, ever.

This surprises people, so the reasoning is worth being explicit about. LLM Metrix can scan a competitor’s brand the same way it scans yours; that is how competitor benchmarking works. But a competitor scan produces a competitor’s visibility score, and emailing you “Acme Corp AI visibility 62/100” as though it were an alert about your own brand is not a notification, it is a category error. Worse, it is the kind of category error that reads as perfectly plausible for months before someone notices the number in their inbox was never theirs.

So the notification builder checks the scan’s subject type and returns zero channels for anything that is not a brand scan. Competitor data lives on the competitor surfaces, where it is labelled as what it is.

Sending the same alert twice is worse than sending it late

Scans run through an asynchronous job queue, and that queue is at-least-once by design: if a worker dies mid-job or a message is redelivered, the job runs again. That is the right trade for correctness (losing a scan is worse than repeating one), but it means a naive notification step would email you twice for a single scan.

Every notification job therefore carries a deduplication key scoped to the scan and the channel. Inserting a job that already exists is a no-op. A retried scan worker can call the notification path as many times as it likes; you get one email, one webhook POST.

This is unglamorous, and it is the single most important property in the whole feature. Duplicate alerts are how people learn to ignore a channel.

When we stop emailing you entirely

Delivery feedback is wired in, and it changes behaviour rather than merely being logged.

When the email provider reports a hard bounce (the mailbox does not exist) or a spam complaint, the address goes onto a suppression list and future sends to it are skipped. The two are treated differently. A hard-bounce entry carries a finite expiry, so a mailbox that was mis-provisioned and later fixed recovers on its own without anyone running production SQL. A spam complaint never lapses, because someone told us explicitly to stop.

Two details matter here.

First, a bounce does not mark the notification job as failed. A bounce is a delivery outcome, not a transient error. Re-driving the job would just re-send to a dead address, and keep doing so until the retry budget ran out.

Second, when a send is skipped for suppression, we log the drop. That sounds too obvious to mention. It was not: a suppressed address produces no job row at all, so an unlogged skip is completely invisible. The user simply stops receiving mail (welcome emails, workspace invites, digests) and nothing anywhere records why. That was a real bug in this codebase, and the fix was one log line. It is a good illustration of a general principle: a system that silently declines to act needs to say so somewhere, or nobody will ever be able to debug it.

Every alert and digest email also carries RFC 8058 one-click unsubscribe headers (List-Unsubscribe and List-Unsubscribe-Post), signed so an unauthenticated click can be trusted without a session. If someone wants out, their mail client’s own unsubscribe button works, and it works in one click.

The two channels

Email delivers the digest described above: the score, the alerts, and a link into the dashboard.

Outbound webhook: a structured JSON POST carrying the event name, brand, domain, score, summary, per-engine results, alerts, and timestamp. This is the one to use if you want alerts in a system we do not integrate with directly.

Each channel degrades independently. An unreachable webhook endpoint does not stop the email.

Setting it up

1. Go to Settings → Notifications in your dashboard.
2. Turn on the channels you want (email and/or an outbound webhook).
3. Choose your email digest frequency: after each scan, daily, weekly, monthly, or off.
4. Leave alert emails on unless you have a specific reason not to. It is what makes a weekly cadence safe.
5. Scan the Alert types switches below. All six are on by default, and that is the right starting point. Turn one off only once it has proved noisy for your brand, since the useful ones differ by how contested your category is. Switching a type off stops it reaching your channels; it stays visible on the Alerts page either way, so nothing is hidden from you, only from your inbox.

Alerting is only as useful as the scan cadence underneath it, and cadence is set by your plan: every paid plan re-scans daily, while the free tier refreshes weekly. Anything tighter than that is a scan you run yourself. If you are deciding how often you actually need to look, how often should I monitor AI visibility works through the trade-off honestly.

The limits, stated plainly

This is per-scan alerting, not real-time monitoring of the engines themselves. An alert fires when a scan completes and finds something, which means detection latency is your scan cadence, not zero. Be concrete about what that cadence actually is: the automated re-scan runs daily on every paid plan, and weekly on free. So if you leave it to the schedule, a problem that appears right after a scan surfaces about a day later on a paid plan (up to a week on free). Running a scan by hand is what closes that gap, and there is nothing stopping you doing it more often: the only limit on an on-demand scan is a one-minute cooldown per subject and your credit allowance.

That is why this post is titled per-scan rather than real-time: the word “real-time” would be doing work the mechanism cannot, and we would rather say so.

There is also no anonymous watching. Alerts attach to a project inside a workspace. There is no way to hand us a domain from a logged-out page and have us monitor it, because running recurring AI queries against a domain nobody signed up for is not a feature, it is an unbounded bill.

Within those limits the point stands: every alert you receive should be worth reading. If you find yourself routinely ignoring one, that is a defect in our design rather than in your filtering, and it is worth telling us about.