Blocking AI Crawlers Does Not Protect Your Content. It Removes You From the Answer.

At some point somebody will suggest you add a few lines to robots.txt to keep AI companies away from your website. It is presented as basic hygiene, the digital equivalent of locking up at night.

It is a real decision with a real cost on both sides, and it deserves more than a copied snippet. The most common version of the advice quietly assumes that all these crawlers are doing the same job. They are not.

Two jobs, one file

Broadly, an AI operator's crawler is doing one of two things.

Collecting text for training. Bulk gathering of pages to build the corpus a future model learns from. This happens ahead of time, at scale, unrelated to any particular user.

Fetching a page because someone just asked something. A person asks an assistant a question, the system decides it needs current information, and it retrieves pages right then to ground its answer — and typically to cite them.

These often use different crawler identities, and operators generally document which is which. Blocking is not one switch. It is a decision you can make separately for each.

What each choice actually costs

Blocking the training crawler stops your future pages being added to future training data. It does not remove anything already collected — that has happened, and a robots.txt change is not a retraction. So the benefit is forward-looking and partial.

Blocking the retrieval crawler is the consequential one. If the system cannot fetch your page at the moment someone asks a relevant question, you are not in the answer. Not described unfavorably — absent. And since retrieval-grounded answers are where citations and the occasional click come from, you have removed yourself from the only part of this that can send you a visitor.

People frequently do the second while intending only the first.

Robots.txt is a request

Worth stating plainly: robots.txt is a convention, not a control. It is a text file asking politely. Well-behaved crawlers honor it. It is not a technical barrier, and it is not access control. If you have material that genuinely must not be read by anyone you have not authorised, the answer is authentication, not a disallow line.

This matters because "we blocked the AI crawlers" is sometimes said in a tone implying content is now protected. It is not protected. It is unlisted, from the ones that comply.

The calculus is not the same for everyone

A newspaper whose business is people paying to read its articles has an obvious interest in not having those articles summarized for free. Several large publishers have negotiated licensing arrangements rather than simply blocking, which tells you they consider the asset valuable enough to price rather than withhold.

A plumbing company in San Antonio is in an entirely different position. Its website is not the product; it is the advertisement for the product. Being summarized, quoted and named in an answer is the thing it wants. Blocking retrieval to protect service-page copy that exists to be found is a straightforward own goal.

Most businesses we work with are the second kind and have occasionally been handed advice written for the first.

A reasonable default

For a business whose site exists to win customers:

Allow retrieval crawlers. You want to be in the answer; that is the entire point of publishing.

Decide about training crawlers on your own view. There is a legitimate argument either way, and the practical stakes for a local service business are modest in both directions. If it matters to you on principle, block them — just know which lines do which.

Never block by guessing. Look up the current documented crawler names from the operator's own documentation before you write a rule. These change, and a rule aimed at a name that no longer exists is doing nothing while you believe it is doing something.

And check what you already have. We regularly find robots.txt files carrying rules somebody pasted in years ago for reasons nobody at the company can now explain, occasionally blocking something the business would very much like to be in.

Next steps

Open yourdomain.com/robots.txt and read it. If there is a rule in there you cannot account for, that is worth ten minutes.

If you would like us to look at what your site is currently telling crawlers — all of them, not just the AI ones — that is part of the free site audit. No obligation, and we will tell you if the answer is "this is fine," because it often is.