In the root directory of almost every website sits a file called robots.txt. It tells automated visitors which parts of the site they may and may not read. For years, that was mostly about Google's crawler. Since AI assistants started reading the web, new names have appeared at the door, and the choice you make in that file determines whether those assistants can mention you.

01 / the trade-offWhat a block actually does

A block does two things at once, and you can't separate them. You prevent your texts from being used to train language models, and you prevent those same systems from retrieving your pages when a user asks something you have the answer to.

That second consequence is often overlooked. Anyone searching via an AI assistant for what you offer gets an answer with sources attached. If you're on the blacklist in robots.txt, your page can't be that source. The answer still comes, just with your competitor in it. Why that happens is explained in more detail in why does ChatGPT mention your competitor but not you.

A crawler that isn't allowed to read your site can't cite you anywhere. The block that protects your content makes you invisible at the same time.

02 / the namesWhich bots are at your door

Each platform has its own crawler with its own name, and your robots.txt treats them by name. These are the bots that matter:

The bottom row is the odd one out. CCBot, the Common Crawl crawler, builds a public copy of the web that mainly serves as training material. CCBot doesn't yield citations; the other bots in the table do, and that's where the advice per row comes from.

For Claude there's an added twist: Claude searches the web via Brave Search, so alongside ClaudeBot, your presence in Brave's index also counts. How that works is explained in does your business appear in Claude.

03 / the middle groundKeeping out training crawlers, allowing search bots

The middle ground is simpler than it sounds: refuse the bots that only collect training material, and let through the bots that provide answers with sources. In practice that means blocking CCBot and allowing the names in the table above.

That distinction isn't entirely clean. What a platform does with retrieved pages is decided by the platform itself, and the line between searching and training shifts by provider and by year. So the middle ground is no guarantee that your texts will never end up in training data. It's a trade-off: closing off most of the training flow while keeping the route to citations open.

Best for: service providers and other businesses whose texts are a shop window rather than a product. For that group, the middle ground is the logical choice: the training flow largely closed off, the route to citations open.

04 / the checkHow to read your own robots.txt

Checking takes two minutes and requires no technical knowledge. Type your domain name into the browser followed by /robots.txt, so for example yourbusiness.nl/robots.txt. You'll see a text file with lines starting with User-agent and Disallow.

Look in that file for the names from the table. If there's a block like "User-agent: GPTBot" followed by "Disallow: /", that bot is fully blocked. If you don't find any of the names, what applies is whatever is listed under "User-agent: *", the rule for all visitors at once.

The outcome regularly surprises. Some site builders and security packages blacklist AI bots by default, without the owner ever having made that choice. Anyone who has never checked doesn't know what choice has been made on their behalf.

05 / the exceptionWhen blocking does make sense

There are situations where fully blocking is the right choice, and then there's nothing further to do here. If you earn directly from your content, for example through paid articles, a knowledge base behind a login, or course material, every free pass-through via an AI answer is a leak in your business model. The same goes for publishers negotiating reuse with AI companies: open doors weaken your negotiating position.

For a service provider it's the other way round. Your texts aren't a product but a shop window, and a citation in an AI answer is exactly what that shop window is meant to do. In that case, a block costs visibility and delivers nothing worth protecting.

This text was produced with AI assistance and checked and approved by a human before publication.