Mostly, yes — the major, reputable AI crawlers honor robots.txt. OpenAI’s GPTBot, Google’s Google-Extended, Anthropic’s ClaudeBot, and PerplexityBot all document the directives they follow. But the picture has nuance: different bots serve different purposes, compliance is voluntary, and blocking a crawler has real visibility consequences. Here’s what you need to know.
How robots.txt works for AI crawlers
robots.txt is a voluntary standard: it tells well-behaved bots which paths they may access. Reputable AI companies publish named user agents and respect Disallow rules for them. For example, you can allow or block each bot independently:
# Block OpenAI's training crawler
User-agent: GPTBot
Disallow: /
# Allow Perplexity's crawler
User-agent: PerplexityBot
Allow: /
The key nuance: “voluntary” means compliance depends on the operator. Major, named crawlers honor it; obscure or bad-actor scrapers may not. See the AI crawlers guide for the current list of user agents.
Training vs. retrieval crawlers
Not all AI bots do the same thing, and the distinction matters for your decision:
- Training crawlers (e.g., GPTBot, Google-Extended, ClaudeBot) gather data that may inform model training. Blocking these limits whether your content shapes what models learn.
- Retrieval/answer crawlers (e.g., PerplexityBot, and search-grounding fetches) pull live pages to answer queries with citations. Blocking these removes you from those engines’ cited answers.
You can make different choices per bot — for instance, allow retrieval crawlers (for citations) while restricting training crawlers, if that fits your strategy.
The visibility trade-off
Blocking AI crawlers is a real lever, but it cuts both ways:
- Block, and you protect content from being used — but you forfeit the AI visibility, citations, and referral traffic that come with being included.
- Allow, and you gain visibility — your content can be retrieved, cited, and represented, at the cost of it being used by AI systems.
Most brands seeking AI visibility should allow the major crawlers and focus on being represented accurately, rather than blocking and becoming invisible. See does AI use my website.
One vendor can operate several bots with different jobs
The training-versus-retrieval split above is real, but it is not one bot per company. OpenAI’s published bot documentation lists distinct user agents for gathering data that may inform training, for building the search index behind ChatGPT’s search results, and for fetching a page live in response to a specific user’s request in a conversation. Each is controlled separately.
This is where most self-inflicted damage happens. A Disallow: / under a wildcard User-agent: *, or a block aimed at the training crawler that also catches the search crawler, removes you from cited answers by a company you only meant to opt out of training with. The three decisions are genuinely separable — read the vendor’s own documentation for the current agent names before writing the rules, and re-read it after any migration.
The rule that surprises people: precedence, not accumulation
robots.txt groups do not stack. Under RFC 9309, a crawler picks the single most specific group matching its own name and ignores the rest, so a group naming GPTBot replaces the * group rather than adding to it.
The consequence catches careful people. If your * group blocks /admin and /checkout, and you then add a GPTBot group containing only Disallow: /private, GPTBot is now permitted into /admin and /checkout — the global rules never applied to it. Whenever you add a bot-specific group, restate the rules you actually want that bot to follow.
Two more details from the specification worth knowing: within the applicable group, the longest matching rule wins regardless of order (so an Allow can carve an exception out of a broader Disallow), and a robots.txt that returns a server error should be treated as a complete disallow rather than as permission — which is why a broken deploy of that one file can quietly take you out of AI answers.
User agents are claims, not identities
A User-agent header is a string anyone can send. Scrapers routinely impersonate reputable crawlers precisely because sites trust the name, so your server logs are not evidence of who visited — they are evidence of what someone typed.
The major operators publish either IP ranges or support reverse-DNS verification for this reason: you resolve the requesting IP, confirm the hostname belongs to the operator’s domain, and resolve it forward again to confirm it matches. Anything that fails that check and claims to be a well-known bot is something else.
This matters for two decisions. It tells you whether your Disallow rules are actually being honoured by the bots you meant them for, and it stops you concluding that a crawler is hammering your site when the traffic is an impersonator. AI traffic analytics performs that verification and separates confirmed crawler visits from unverified ones, which is the difference between measured crawler behaviour and a log full of assertions.
Practical checklist
- [ ] Review
robots.txtfor unintended blocks of AI user agents - [ ] Decide per-bot: allow retrieval crawlers if you want citations
- [ ] Confirm key content isn’t hidden behind scripts retrieval can’t read
- [ ] Consider an llms.txt to guide AI to your best content
- [ ] Re-check after site migrations, which often reset crawler rules
Frequently Asked Questions
Do AI crawlers respect robots.txt?
The major reputable ones do — GPTBot, Google-Extended, ClaudeBot, and PerplexityBot all document and honor robots.txt directives. Compliance is voluntary, however, so obscure or bad-actor scrapers may ignore it.
How do I block AI crawlers?
Add Disallow rules for the specific user agents in your robots.txt (for example, User-agent: GPTBot then Disallow: /). You can block or allow each bot independently depending on your strategy.
Should I block AI crawlers?
Usually not, if you want AI visibility. Blocking removes you from AI answers, citations, and referral traffic. Most brands benefit from allowing the major crawlers and focusing on being represented accurately instead.
What’s the difference between training and retrieval crawlers?
Training crawlers gather data that may inform model training, while retrieval crawlers fetch live pages to answer queries with citations. Blocking training crawlers limits your influence on what models learn; blocking retrieval crawlers removes you from cited AI answers.