AI Crawlers: Should You Block GPTBot? Every Token, and What Blocking Really Does

The GPTBot block most sites got wrong

OpenAI runs three crawlers with three jobs. GPTBot collects content to train OpenAI's foundation models. OAI-SearchBot finds pages to show in ChatGPT's search answers. ChatGPT-User fetches a page when a person in ChatGPT asks it to. OpenAI says blocking GPTBot is a separate control from OAI-SearchBot, and that sites which opt out of OAI-SearchBot will not be shown in ChatGPT search answers.

So a site with 'User-agent: GPTBot' and 'Disallow: /' has opted out of training and nothing else. A site that blocked every OpenAI token has opted out of being recommended by ChatGPT search, which is a strange thing for a plumber to want. The same split now exists at Anthropic, Apple, Amazon, Meta and Google, and the same confusion follows it everywhere.

Training, search and fetch: three different jobs

Before the table, learn the three categories. Every AI token falls into one of them, and the category tells you what blocking it costs.

All 23 AI crawler tokens, side by side

This is every token we could verify on the operator's own documentation (or, where noted, credible reporting), as of October 2026. 'Blocking it' means a Disallow rule in robots.txt for that token.

Twenty-three tokens, and only two of them (Googlebot and Bingbot) are crawlers most sites already know. If you read one row twice, read OAI-SearchBot's. It is the switch that decides whether ChatGPT search can recommend you at all. The engine pages for Perplexity, Claude and Apple Intelligence and Siri cover what each engine does with what its crawler finds.

Three robots.txt policies you can copy

A crawler follows the group in robots.txt that names it, and falls back to the 'User-agent: *' group only when none does. Several User-agent lines can share one rule. Each item below is one line of the file, in order, so you can paste them as written.

Policy 1: every crawler welcome

• User-agent: *
• Allow: /
• Sitemap: followed by the full address of your sitemap, so every crawler can find your pages.
• Note: an empty robots.txt does the same job. This is the right default for most local businesses that want to be found.

Policy 2: answers yes, training no

Policy 3: AI engines out, classic search in

• Start with the first group from Policy 2, then add these User-agent lines to it: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User.
• Add Google-Extended, Google-CloudVertexBot, Meta-WebIndexer, Meta-ExternalFetcher, Amazonbot, Amzn-SearchBot and Amzn-User to the same group.
• Close the group with Disallow: / on its own line.
• Then a blank line, User-agent: * and Allow: /, so Googlebot, Bingbot and Applebot still index you for classic search.
• To leave Google's AI Overviews and AI Mode too, use the Search Console 'Search generative AI' control. For Copilot, use NOARCHIVE. Robots.txt cannot do either without dropping you from search.
• Expect leaks: ChatGPT-User, Perplexity-User, Meta-ExternalFetcher and Amzn-User are user-triggered fetchers whose operators say robots.txt may not apply.

Which policy a business that sells something should pick

If customers find you by asking questions, block nothing that answers questions. Policy 1 or Policy 2 is right for nearly every contractor, retailer, practice and firm we work with. The choice between those two is about values and licensing, not visibility: do you want your words in the next model's training data? There is no verified evidence that opting out of training changes whether an engine recommends you today, because recommendations come from the search crawlers and the live web.

Policy 3 makes sense for publishers who sell their words, for paywalled research, and for firms negotiating licensing deals. For them, an AI answer that summarizes the article is a lost subscriber. For a roofer, an AI answer that names the roofer is a lead. Same file, opposite economics.

Robots.txt is a sign on the door, not a lock.

That is the honest limit of all three policies. Well-behaved crawlers obey. User-triggered fetchers may not. Cloudflare reported in August 2025 that Perplexity used undeclared crawlers with changing user agents to get around no-crawl rules, and removed it from its verified bots list. Anthropic warns that blocking its IP addresses can backfire, because the bot then cannot read robots.txt at all. If enforcement truly matters, it happens at the CDN or firewall, checked against the IP lists OpenAI, Perplexity and Common Crawl publish.

Audit your own robots.txt in ten minutes

Sites can block AI search by accident: an overcautious plugin, a hosting default, a line copied from a forum years ago. Here is how to find out what yours says.

1. 1

   Open the file

   Type your domain followed by /robots.txt into a browser. If it returns nothing or a 404 page, every crawler is allowed.

2. 2

   Look for a blanket block

   A 'User-agent: *' group with 'Disallow: /' blocks everyone, Googlebot included. If you find it on a live site, see robots.txt blocking Google right away.

3. 3

   List every named token

   Write down each User-agent line and compare it with the table above. Note any search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) that is disallowed.

4. 4

   Check each subdomain

   Anthropic says ClaudeBot rules must be set for each subdomain. A shop or booking subdomain has its own robots.txt.

5. 5

   Check the layer above the file

   Ask your host or CDN whether an AI-bot blocking setting is switched on. It can refuse crawlers before robots.txt is ever read.

6. 6

   Confirm in the answers

   After any change, allow a day or so (OpenAI says about 24 hours for its systems), then run your customers' questions through the free AI visibility check.

Crawl access is the first thing a GEO audit checks, before content or reviews, because nothing else matters if the engines cannot read the site. Even then, a crawler that can read the file may not see the content: most AI crawlers do not run JavaScript, which is its own problem, covered in can AI crawlers read JavaScript.

Crawlable is the start, not the finish

Opening the door does not get you recommended. It gets you considered. After access come the things that decide whether the engine picks you: a consistent entity record, reviews, mentions on pages the engines trust, and content written so a model can lift a sentence intact. The GEO guide lays out that whole road.

DataCram's GEO and AI visibility programs start with crawl access and end with monthly reports that show the actual answers. We fix the robots.txt and server setup so crawlers can read the site, then do the slower work of listings, reviews and quotable content. Programs start at $625 a month with no long contract, set after a free audit, as explained on GEO pricing.