Blocking AI crawlers can be an editorial decision, a response to server load, or a position on how your material gets reused. A blanket block, though, treats very different activities as if they were one thing: collecting pages that may feed model training, fetching sources for an AI search index, and reading a page because someone asked a question about it. The right decision depends on what you accept, what you want to keep visible, and what you can actually enforce.
Allowing a crawler guarantees nothing. Not a citation, not a visit, not a customer. Blocking one does not undo an earlier collection, and it does not remove what other sites already publish about your organisation. Both directions are choices with limits, and both are worth writing down before you touch the file.
What follows is a way to build a policy you can explain to a colleague, apply per content type, and verify on the live site rather than on assumptions.
Identify what each crawler does before you write a rule
The names look alike. The functions and the published rules are not interchangeable. The table below follows the distinctions the providers themselves document at the time of writing; check the source pages before you act, because these lists change.
| Provider | Collection that may feed training | Search or user-requested reading |
|---|---|---|
| OpenAI | GPTBot collects content that may be used for models. | OAI-SearchBot relates to search; ChatGPT-User to certain user-triggered fetches. |
| Anthropic | ClaudeBot covers collection that may contribute to training. | Claude-SearchBot contributes to search; Claude-User reads pages on request. |
| Perplexity | PerplexityBot is not presented as a foundation-model training crawler. | PerplexityBot serves search; Perplexity-User answers user requests. |
According to OpenAI's crawler documentation, GPTBot and OAI-SearchBot are configured independently. Disallowing OAI-SearchBot takes your pages out of ChatGPT search answers, with a possible exception for navigational links. ChatGPT-User does not govern that eligibility, and robots.txt rules may not apply to the fetches it makes on a user's behalf.
Anthropic documents robots.txt handling for its three agents. Perplexity, for its part, states that its user-triggered agent generally ignores those rules. So read the rule each service publishes for itself. Do not infer one agent's behaviour from a competitor's.
Google: crawling, training and AI answers are separate switches
Google-Extended is a control token, not a separate HTTP user agent you will find in your logs. It covers Gemini training uses and the grounding of answers in Gemini Apps and Vertex AI. Google states that it affects neither inclusion nor ranking in Search. The official crawler list sets out that scope.
A second distinction matters as of September 2026: Search Console now carries a control specific to the generative features of Search. Google says its worldwide rollout completed on 31 August 2026. It lets you keep links and content out of the covered features, including AI Overviews and AI Mode, without acting as a ranking signal for the rest of Search. It does not govern training. The documentation explains parent properties and how long a change takes to apply.
The claim that you must leave Google Search entirely in order to decline AI answers is therefore out of date. The reverse holds too: blocking Googlebot is a far wider decision than declining those particular features, and it is rarely the one you meant to make.
Which policy suits your business?
For a services firm, the question is often whether accurate commercial information should stay readable while certain training uses are declined. For a publisher, the value may sit in reading on your own site, in subscriptions, or in reuse rights that are negotiated rather than assumed. For a shop, interrupting access to the catalogue can break journeys that were doing useful work. None of these situations automatically justifies opening everything or closing everything.
A more workable approach is a decision per content type: presentation pages, catalogue, articles, research, documentation, customer area. Write down who can overrule the default, why each exception exists, and how to revert. And treat the argument that training exposure will make your brand better known for what it is: a hypothesis, not a demonstrated route to recommendation or sales.
| Objective | Control to examine | Limit |
|---|---|---|
| Decline collection for training | The provider's own token and its documented rules. | Not a retroactive deletion of copies already made. |
| Keep pages readable by an AI search engine | The search agent, the firewall, and whether pages render. | Being reachable guarantees no selection. |
| Protect a confidential document | Authentication and access permissions. | robots.txt is not an access control. |
| Reduce excessive load | Request measurement, server rules and sensible limits. | A rule drawn too broadly can obstruct legitimate visitors. |
Load deserves its own measurement rather than a guess. If crawler traffic is genuinely degrading response times, the evidence sits in your logs and in your site speed measurements, not in the reputation of a particular user agent.
A deliberately narrow example, to merge into the file you already have
If your decision is only to decline the GPTBot and ClaudeBot collections, the groups below illustrate that choice. They are not a complete robots.txt, and they say nothing about the other agents.
# Example: two collections declined
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Before adding them, read the groups already in the file. Rules pasted in at random can make it contradictory, or quietly change the behaviour of a named crawler. Keep the technical exclusions that earn their place, keep the real sitemap reference, and avoid the universal template that explicitly allows Googlebot every path without anyone checking what the existing restrictions were for.
The specification as Google interprets it separates agent groups and limits a file's reach to its own host, protocol and port. A file on the main domain does not automatically govern every subdomain. The syntax has to stay plain text, too: an HTML page that displays the rules attractively is not the file a crawler is asking for.
robots.txt does not protect a private area
The file is public. Writing the path of a confidential folder into it does not make that folder inaccessible; it advertises the path. Google also notes that a URL disallowed from crawling can still appear in results when it is known from elsewhere. Private data needs an access control. Removal from an index needs an indexing mechanism built for that purpose. These limits are set out in the introduction to robots.txt.
A firewall or a CDN may also act before a crawler ever reaches WordPress. A permissive file does not cancel an HTTP refusal, a JavaScript check or a CAPTCHA, which is the usual explanation when a site is allowed on paper and blocked in practice. And do not authorise a sensitive request on the strength of a user agent string alone: names can be spoofed. Where a provider publishes IP ranges or a verification method, use those, with narrow rules.
Check the effects, and resist quick conclusions
- Archive the starting state. Save the file, the relevant settings, and a few representative HTTP responses so you can compare later.
- Test real access. Request a public page, a resource it needs in order to render, and one of the paths you intended to restrict. Test at the server and at the CDN, because they can disagree.
- Read the logs you have. Look at requested paths, response codes and verified agents. Absence over a short window does not prove a block.
- Compare after the change. Separate access errors from content problems. A missing citation is not evidence of a technical fault.
An indexing review deals with access and eligibility, which are the two things most often confused here. A technical SEO audit puts those observations back into the general state of the site. Keep the distinction between a verifiable fix and a hoped-for visibility outcome; they belong in different columns of your report.
Do you also need an llms.txt file?
Do not confuse it with robots.txt. Google states that no special AI file and no additional mark-up are required for its AI features. An experimental document describing your content may have a particular internal use, but it does not replace access to your pages or the quality of what is on them, and it must contain nothing private. Google's conditions for sites are published here.
The priority stays the same: a documented choice, consistent with how your business actually earns, tested on the real site. When you scope an audit, it is worth naming the agents to be examined, the access evidence expected, and the limits of the diagnosis, so that nobody reads a technical check as a promise of being quoted.
Common questions
Does blocking GPTBot keep my site out of ChatGPT search?
These are two separate controls at OpenAI. GPTBot concerns collection that may serve training; OAI-SearchBot concerns search. Check the firewall as well: an allowance in robots.txt counts for nothing if the server refuses the request.
Does ChatGPT-User always obey robots.txt?
No. OpenAI indicates that the rules may not apply to actions initiated by a user. That agent does not govern eligibility for search either. Anything genuinely private needs a real access control, not a directive.
Can I stay out of AI Overviews without leaving Google Search?
The Search generative AI control in Search Console now allows that exclusion for the features it covers. Check the setting on the right property and whether it is inherited from a parent property. This control is separate from anything to do with training.
A crawler never appears in my logs. Does that prove the block works?
No. The crawler may simply not have chosen to visit during the period you looked at. Look instead for refused responses, a specific matching rule, or an access inconsistency you can reproduce, before you conclude anything.
Does allowing a crawler guarantee a citation?
No. At best it removes an access obstacle. Which source gets used then depends on the service, the question and what else is available. Observe that separately from your technical tests, and do not let one stand in for the other.
Does the robots.txt on my main domain cover my subdomains?
No. A file governs its own host, protocol and port. A shop or a documentation site on a subdomain needs its own file. The same applies to a staging environment, which usually needs different rules from production.