Cloudflare has launched Bot Preference Sync, a tool that automatically generates and updates your site's robots.txt file based on rules you set by AI crawler category, not bot by bot. You can, for example, allow OpenAI's crawlers and block ByteDance's with a single category choice, without writing a line of code.
If you run a company website, an e-commerce store or an industry blog, this changes something concrete: your visibility in AI results (ChatGPT, Perplexity, Google AI Overviews) also depends on which crawlers you let through. Until now, keeping that list up to date was manual work, and often neglected. From now on, Cloudflare can do it for you.
If your site already uses Cloudflare as a CDN or firewall, it's worth exploring this feature this week. If you don't use it, the article still explains the logic, so you can update your robots.txt by hand with a clear method.
The problem nobody wants to face: your robots.txt is stuck in 2019
The robots.txt file is a list of instructions that tells bots (search engines, AI crawlers, assorted spiders) what they can read on your site and what they can't. It has been around for decades and works well, but it needs manual updates every time a new crawler hits the market.
In 2026, AI crawlers multiply every quarter. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Bytespider (ByteDance), CCBot (Common Crawl), and more on the way. Each crawler has a different name in robots.txt. Who keeps that list current? Almost nobody.
The typical result: a robots.txt configured years ago, AI crawlers coming and going at random, and no real strategy on who can index your content for AI answers. If you're trying to understand how AI visibility works as a metric for SMEs, robots.txt is one of the first places where you lose ground without noticing.
What Cloudflare's Bot Preference Sync does
Cloudflare has unveiled Bot Preference Sync: a tool that automatically generates and updates your domain's robots.txt, based on rules you set by crawler category, not by individual bot.
The logic is simple: instead of writing one line for GPTBot, one for ClaudeBot, one for PerplexityBot and so on, you choose a position for each category ("AI crawlers for training", "AI crawlers for real-time answers", "traditional search engine crawlers"), and Cloudflare turns that choice into specific rules, updated whenever new bots are added to its database.
As reported by Search Engine Journal, the key point is that no single setting fully describes what a company wants to do: allowing GPTBot and blocking Bytespider are two different choices, with different motivations, and a category-based tool makes them manageable without advanced technical skills.
Three steps to use it, or to do it by hand if you don't use Cloudflare
If your site is already on Cloudflare:
- Log in to your domain's Cloudflare dashboard.
- Find the Bot Management section (available on Pro plans and above) and check whether Bot Preference Sync is already active or rolling out for your account.
- Set your preferences by category: decide which AI crawlers you want to allow and which to block, based on how they use your content.
If you don't use Cloudflare:
- Open your robots.txt (usually at yourdomain.com/robots.txt).
- Manually add directives for the main AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot. Each crawler has a specific name to use in the line
User-agent. - For each one, decide whether you want to
Allow: /oDisallow: /, based on your AI visibility strategy.
The difference between the two approaches is maintenance: with Cloudflare, the list updates itself. By hand, you have to keep an eye on it every time a relevant new crawler shows up.
Who really needs this feature, and who can wait
You need it right away if you:
- publish original content (guides, catalogs, industry articles) that you want cited in AI answers;
- have already worked on a content validation workflow for AI Search and want the right crawlers to reach it;
- run an e-commerce store or a site with sensitive data that you don't want certain models to train on.
You can wait if you:
- have a brochure site with few updates and no active content strategy;
- don't use Cloudflare yet and don't have the resources to migrate right now.
Careful: blocking all AI crawlers out of fear is not a strategy. It means dropping out of ChatGPT, Perplexity and Google AI Overviews results. If visibility in AI answers is already a goal in your marketing plan, blocking the crawlers that feed it is an own goal.
The typical mistake and what this tool doesn't do
The most common mistake is treating robots.txt like an on/off switch: either you block everything or you let everything through. Cloudflare's category logic pushes you to think more granularly, and that's the real value of the feature.
What Bot Preference Sync doesn't do:
- It doesn't guarantee that crawlers will follow the instructions. Robots.txt is a voluntary protocol: legitimate bots follow it, malicious ones don't.
- It doesn't replace a content strategy. Opening the door to AI crawlers doesn't mean getting cited: your content has to be authoritative, structured and relevant. On this, it's worth reading how local visibility in AI Search works for Italian SMEs.
- It doesn't solve the problem of conflicting brand information already circulating in AI datasets.
Robots.txt is one piece of a broader SEO and AI visibility strategy. Keeping it up to date is necessary, but not sufficient.
Related articles
Discover more articles exploring similar topics, selected to offer you a more complete and stimulating perspective. Each piece of content is carefully chosen to enrich your experience.
