Should You Block AI Crawlers? The Trade-Off Nobody Quantifies

“Should we block AI crawlers?” is asked as one question and is actually three. The single most expensive mistake in this area is answering all three at once with a blanket robots.txt rule — because the bots involved do very different jobs, and blocking them costs you very different things.

Several publishers have made this error in a specific and painful form: intending to stop their archives being used for model training, they blocked broadly, and in doing so also removed themselves from the retrieval index that grounded answers cite. They gave up the visibility and kept none of the leverage.

Here is the decomposition, and an honest account of what each choice actually buys.

Three bots, three bargains

Training crawlers collect content for future model training runs. GPTBot is documented in this role in OpenAI’s crawler documentation; ClaudeBot similarly in Anthropic’s crawling FAQ.

What blocking costs you: very little that is measurable in the near term, and something real in the long term. Training influence is what makes a model know who you are without looking you up. Blocking forfeits that for future model generations — a slow, unobservable cost you will never be able to attribute.

What blocking buys you: your content is not used to train that vendor’s future models. Whether that is worth anything depends entirely on whether you have licensing leverage. For a major publisher with a distinctive archive, it is a negotiating position. For most companies, it is a principle with no counterparty.

Search/index crawlers build the retrieval index that grounded answers draw from. OAI-SearchBot sits here, as do Perplexity’s crawlers, documented in Perplexity’s crawler reference.

What blocking costs you: this is the expensive one. Blocking here means you are not in the candidate pool when someone asks a question your content answers. Not deprioritised — absent. And the loss is immediate rather than deferred.

What blocking buys you: almost nothing, for almost anyone. This is the bot that sends you citations and referral traffic. OpenAI’s publishers and developers FAQ documents the separation between the training and search crawlers precisely so that site owners can make this choice independently.

User-triggered fetchers fire when a specific person asks a question and the assistant fetches a page to answer it. ChatGPT-User is the clearest case.

What blocking costs you: a real human, right now, wanted an answer and the assistant tried to read your page to give it to them. Blocking this is the closest thing in the taxonomy to declining an inbound visit.

What blocking buys you: nothing beyond bandwidth. There is no training use and no index; it is a fetch on behalf of a user.

The decision, for most companies

If you are a software company, a services business, an ecommerce brand, a B2B vendor — anything whose business is not selling access to a content archive — the answer is straightforward and worth stating plainly:

Do not block search crawlers or user-triggered fetchers. These are your distribution. Blocking them to protest training is like removing yourself from Google’s index to protest ad targeting: costly, symbolic, and aimed at the wrong mechanism.

Training crawlers are a judgement call with low stakes either way. Allowing them is a small bet that being in future training corpora helps. Blocking them is a small principled stand with a slow, unmeasurable cost. Neither will decide your quarter.

The asymmetry is what matters: the search-crawler decision has a large, fast, observable effect and the training decision has a small, slow, unobservable one. Spending your deliberation on the second while getting the first wrong by accident is the actual failure mode.

When blocking is genuinely defensible

Three cases, and they are narrower than the discourse suggests.

Your content is the product. Subscription research, proprietary datasets, paywalled journalism. If someone can get the substance of what you sell from an AI answer, the summary substitutes for the purchase. This is a real business conflict and blocking is a rational response — though it is worth being honest that it also removes you from the discovery surface where new subscribers increasingly begin. AEO for publishers is where this trade-off is sharpest.

You have licensing leverage and intend to use it. Blocking is a negotiating position only if there is a plausible negotiation. Several large publishers have converted this into paid agreements. This requires an archive a vendor actually wants.

Legal or contractual constraint. Licensed third-party content, regulated material, contractual restrictions on redistribution. Not a strategy question.

Notice what is not on the list: “we don’t want AI using our content” as a general sentiment. That is a coherent position and it is not a business case, and it should be taken deliberately rather than arrived at by pasting a robots.txt snippet from a blog post.

The mechanics, and their limits

Control is per-user-agent in robots.txt, which is what makes the three-way split actionable — you can allow the search crawler and disallow the training crawler with two directives. The convention is formalised as RFC 9309, the robots.txt glossary entry covers syntax, and do AI crawlers respect robots.txt covers compliance in practice.

Three limits are worth knowing before you rely on it:

It is voluntary. RFC 9309 is a convention that well-behaved crawlers honour. The major documented AI crawlers do. Scrapers that do not identify themselves honestly are unaffected by any of this, and if your concern is content theft rather than vendor policy, robots.txt is the wrong instrument entirely.

Agent names change. New crawlers appear; existing ones get renamed or split. A rule written two years ago covers a fleet that no longer exists. This needs periodic review, and the only reliable way to know what is actually hitting you is to look — see what your server logs say about AI crawlers.

Blocking does not remove what is already learned. Content already in a training corpus stays there. Blocking is prospective only, and a model that formed an opinion of you in 2024 will continue to hold it.

The measurement gap that makes this hard

The reason this debate is unusually noisy is that the two sides of the trade are measured with wildly different precision.

The cost of blocking search crawlers is observable: citations stop, referral traffic falls, your absence shows up in any monitoring you run. The benefit of blocking training crawlers is unobservable in principle — there is no way to detect whether your content influenced a model, and no counterfactual model to compare against.

So one side of the ledger produces a chart and the other produces a conviction. That does not make the conviction wrong, but it should make you suspicious of anyone presenting this as an obvious call in either direction. If you do restrict something, monitor what happens to your presence afterwards — multi-engine monitoring at least makes the observable half observable, and knowing you dropped out of an engine’s answers is better than inferring it from a traffic dip six weeks later.

The summary

Split the decision. Allow user-triggered fetchers — that is a person asking. Allow search crawlers unless your content is literally the product you sell. Decide on training crawlers according to whether you have leverage worth exercising, and accept that you will never measure the outcome.

And whatever you decide, write it down somewhere with the reasoning attached, because in eighteen months somebody will find the robots.txt rule, not remember why it exists, and either remove a deliberate policy or preserve an accidental one.