- Is blocking AI bots really worth it, or do I risk losing visibility?
- How many types of AI bots are there and what do they actually do on my site?
- How much has AI bot traffic on my site grown over the past year?
- What's changing between training bots and retrieval bots: the real data
- Is robots.txt still enough to manage AI bots?
- Three concrete scenarios: what changes for an e-commerce, a professional firm, and a manufacturing company?
- How do you build a bot policy that really works?
- Why everyone is talking about it now: is the market really accelerating?
- What are the most common mistakes in managing AI bots?
- How do I know if my AI bot strategy is working?
- So, is it worth building an AI bot policy for your company?
- How we see it at SHM
- Frequently Asked Questions
- Blocking all AI bots indiscriminately reduces traffic: publishers who did it lost 23% of their monthly visits.
- There are at least three types of AI bots with different purposes: training, retrieval and autonomous agents. Treating them all the same is a costly mistake.
- robots.txt alone isn't enough: about 30% of AI crawlers ignore it. You need a CDN, firewalls and a coordinated policy across SEO, IT and legal.
- The share of crawling for retrieval and user actions has grown almost fourfold in a year: the trend is accelerating, and waiting has a cost.
Sites that blocked AI crawlers via robots.txt lost an average of 23% of their monthly visits. Yet 86% of the most visited websites have not written a single rule for AI bots.
Is blocking AI bots really worth it, or do I risk losing visibility?
It depends on which bot you block and why. Blocking everything without distinction has already cost traffic for those who did it: the data is clear.
The first question a marketing manager asks is simple: block or don't block? The answer, unfortunately, is not as simple. Blocking a bot that trains an AI model is one thing. Blocking a bot that retrieves content to answer users' questions on ChatGPT or Google AI Overviews is another story entirely: in the second case, you disappear from the AI answers your potential customers are already reading.
A study cited by Search Engine Land, conducted by Rutgers Business School and The Wharton School, measured what happens to publishers that block AI crawlers via robots.txt: a 23.1% drop in total monthly visits and a 13.9% drop in human traffic. It's not a theoretical risk: it has already happened.
| Your situation | Should you block? | why |
|---|---|---|
| You want to protect original content from AI training | Yes | Block only training bots (e.g. CCBot, Common Crawl) |
| You want to appear in AI answers (ChatGPT, Perplexity, AI Overviews) | No, for now | Blocking retrieval bots excludes you from AI answers |
| You have server load issues from excessive crawling | Yes | Limit the frequency (crawl-delay) instead of blocking everything |
| You haven't yet analyzed which bots visit you | No, for now | Map first, then decide: blocking blindly is risky |
| You have restricted or paid content | Yes | Selectively block the protected sections, not the whole site |
How many types of AI bots are there and what do they actually do on my site?
There are at least three categories with very different purposes: training bots, retrieval bots and autonomous agents. Confusing them is the most common mistake.
When people talk about "AI bots," they lump together tools that do completely different things. Understanding the difference is the first step toward making sensible decisions.
| Bot type | What it does | Well-known examples | Impact if blocked |
|---|---|---|---|
| Training bots | Download content to train AI models | GPTBot (OpenAI), CCBot (Common Crawl) | No effect on current visibility |
| Retrieval bots | Fetch content to answer questions in real time | ChatGPT-User, PerplexityBot, Google-Extended | You disappear from AI answers shown to users |
| Autonomous agents | Browse the site like a user to complete tasks (comparisons, purchases) | AI operators, headless browsers | You lose visibility in automated purchasing scenarios |
| Traditional SEO crawlers | Index pages for classic search results | Googlebot, Bingbot | Drop in classic organic rankings |
Autonomous agents, bots that browse your site the way a real user would, are the trickiest case. They can trigger analytics tracking, look like human visitors and become harder and harder to identify. If an agent is comparing your products with your competitors' on behalf of a potential customer, blocking it means you don't exist in that evaluation.
To dig deeper into how AI engines use content to answer users, also read how to adapt your strategy when customers no longer visit your site.
How much has AI bot traffic on my site grown over the past year?
A lot. GPTBot increased its requests by 305% in one year, and bots that simulate human behavior grew 15-fold in 2025.
Cloudflare's numbers show growth that many sites haven't yet spotted in their own logs. GPTBot, OpenAI's crawler, climbed from 9th to 3rd place among all monitored bots in just one year. Overall AI crawling (excluding Googlebot) reached 4.2% of all HTML requests on the Cloudflare network, almost as much as Googlebot alone (4.5%).
The most worrying finding concerns bots that ignore the rules: according to Search Engine Land, in Q4 2025 about 30% of AI crawlers did not respect robots.txt directives, up from an already worrying 3.3% a year earlier. This means that even a perfect robots.txt is not enough on its own.

What's changing between training bots and retrieval bots: the real data
Crawling for retrieval and user actions has nearly quadrupled in a year. The distinction between training and retrieval is no longer theoretical: it's measurable and evolving fast.
Data from TechnologyChecker.io shows how the makeup of AI traffic is changing. Training crawling remains the biggest slice, but crawling for search and user actions, the kind that determines whether you appear in AI answers, grew from 3.14% to 11.92% in less than a year.
In practical terms: the share of AI traffic that decides whether your site appears in ChatGPT or Google AI Overviews answers has nearly quadrupled. Those who blocked everything without distinction lost exactly this visibility. Those who left everything open without control exposed their content to training with nothing in return.
To understand how to measure your current visibility on AI engines, you can start with how to benchmark your AI visibility against competitors.
Is robots.txt still enough to manage AI bots?
No. robots.txt is voluntary: bots respect it only if they choose to. Real enforcement happens at the CDN, firewall and mitigation service level.
The robots.txt file, the text document that tells crawlers what they can and can't visit, is a useful but insufficient tool. It's a good-faith agreement: well-behaved bots follow it, the others don't. In Q4 2025, 30% of AI crawlers ignored it. In Q2 2026, 8.56% of AI requests received a 403 response (access denied), more than double the 3.63% of the previous year: a sign that more and more sites are adding server-level blocks, not just in robots.txt.
| Tool | How it works | Main limitations | Who manages it |
|---|---|---|---|
| robots.txt | Text directives for crawlers | Voluntary, ignored by 30% of AI bots | SEO team |
| CDN (content delivery network) | Filters traffic before it reaches the server | Requires technical setup; can conflict with robots.txt | IT / DevOps team |
| WAF (web application firewall) | Blocks suspicious requests in real time | Can block legitimate bots if misconfigured | IT / Security team |
| Bot mitigation services | Identify and manage advanced bots and agents | Additional cost; requires ongoing monitoring | IT / Security team |
| Company bot policy | Rules shared across SEO, IT, legal and sales | Requires cross-functional coordination | Management + all teams |
The real problem is that the SEO team often writes the rules in robots.txt, while the IT team configures the CDN or firewall differently, and the two systems contradict each other without anyone noticing. You need a single policy, shared and checked regularly.
Three concrete scenarios: what changes for an e-commerce, a professional firm, and a manufacturing company?
The consequences vary a lot depending on the type of site and audience. Here are three illustrative examples to understand what's at stake.
Illustrative examples: numbers are meant to show the reasoning, they are not real customer data.
| Company type | Risk if you block everything | Risk if you block nothing | Recommended strategy |
|---|---|---|---|
| E-commerce (e.g. clothing) | Disappears from product comparisons on ChatGPT and Perplexity; loses sales from AI-assisted users | Content and prices used to train competitors' models | Allow retrieval bots, block training bots, monitor agents |
| Professional firm (e.g. legal consulting) | Not cited in AI answers to frequently asked legal questions | Specialist content used for training without attribution | Block training, allow retrieval with structured markup |
| Manufacturing company (e.g., components) | Technical data sheets that B2B buyers' AI agents can't retrieve | Proprietary technical data exposed to training of third-party models | Public sections open to retrieval, restricted areas blocked |
For a manufacturing company with a complex technical catalog, the stakes are twofold: on one hand, B2B buyers increasingly use AI agents to compare suppliers; on the other, proprietary technical data is exactly the kind of content AI models want for training. The distinction between the two purposes is not a technical detail: it's a business choice.
How do you build a bot policy that really works?
You start by mapping the bots that visit your site, identify the purpose of each one, and build a coordinated set of rules across all the teams involved.
- 1 · SEO + IT teamMap the bots visiting your siteAnalyze your server logs from the last 30-90 days. Identify every AI user-agent present, how often it visits and which pages it hits.
- 2 · SEO teamClassify each bot by purposeTell training, retrieval and autonomous agents apart. Use an up-to-date list of AI user-agents as a reference (e.g. the one from Search Engine Journal).
- 3 · SEO + Legal + Sales teamDefine what you want to protect and what you want to exposeHigh-value original content? Protect it from training. Product pages and guides? Open them to retrieval. Restricted areas? Block everything.
- 4 · IT / DevOps teamImplement the rules at CDN and WAF levelDon't rely on robots.txt alone. Configure your web application firewall to enforce the rules even on bots that ignore robots.txt.
- 5 · All teamsAlign robots.txt, CDN and WAFCheck that the rules across the three systems are consistent. A contradiction between robots.txt and your CDN can block bots you meant to let through.
- 6 · SEO + IT teamMonitor and update every quarterNew bots appear every month. Review your policy at least every three months and after every significant update from the major AI engines.
Why everyone is talking about it now: is the market really accelerating?
Yes. AI crawling for retrieval has nearly quadrupled in less than a year, and the share of sites actively blocking AI bots has more than doubled. The time to decide is now.
Until a year ago, AI crawling was almost entirely dominated by training. Today retrieval, which matters for visibility in AI answers, is growing at a speed no other segment of web traffic has ever shown. At the same time, more and more sites are reacting with server-level blocks: in Q2 2026, 8.56% of AI requests received a 403 response, versus 3.63% in Q2 2025, according to TechnologyChecker.io.
Those who wait for the market to settle risk losing positions that will be hard to win back. Visibility in AI answers is built over time, just like classic organic ranking. To understand how to track your current visibility on ChatGPT and Perplexity, you can read how to track branded and unbranded searches on AI engines.
This topic is part of a broader landscape that we're documenting with regular updates in the SEO and AI visibility cluster.
What are the most common mistakes in managing AI bots?
The most common one is treating all AI bots as a single category. The second is relying only on robots.txt without checking what happens at the server level.
Working with sites of all sizes, the same mistakes keep coming up. Here they are, in order of frequency.
| Error | Why it is a problem | How to fix it |
|---|---|---|
| Blocking all AI bots with a single rule | You lose the retrieval bots that bring visibility in AI answers | Separate training from retrieval in robots.txt and in your WAF |
| Relying on robots.txt alone | 30% of AI bots ignore it; CDN and WAF can override it | Align robots.txt, CDN and WAF with consistent rules |
| Not analyzing server logs | You don't know which bots visit your site or how often | Analyze your logs every month, at least for the main AI bots |
| Not involving the IT and legal teams | SEO decisions get overridden by technical configurations or contractual obligations | Create a bot policy shared across all teams |
| Not updating the rules | New bots appear every month; old rules don't cover them | Quarterly review of the bot list and rules |
A case we see often: the SEO team adds a rule in robots.txt to block a training bot, but the CDN has a configuration that bypasses that rule and lets the bot through anyway. Neither team knows what's really going on. The result is a policy that exists on paper but isn't enforced.
How do I know if my AI bot strategy is working?
Monitor three signals: the share of AI traffic in your logs, your presence in AI answers, and the behavior of organic traffic after every rule change.
| Indicator | What it measures | Frequency | Tool |
|---|---|---|---|
| AI bot traffic in server logs | Which bots visit your site, how often and which pages | Monthly | Log analyzer, Cloudflare Analytics |
| Presence in AI answers | Whether and how often your site is cited by ChatGPT, Perplexity, AI Overviews | Monthly | AEO monitoring tools (e.g. Profound, Bluefish) |
| Organic traffic after changes | Whether the new bot rules have a positive or negative impact on traffic | After every change | Google Search Console, Analytics |
| 403 response to AI bots | How many AI requests are blocked at the server level | Quarterly | CDN dashboard, WAF logs |
| robots.txt / CDN / WAF consistency | Whether the rules across the three systems are aligned | Quarterly | Manual or automated technical audit |
Dedicated tools exist to measure your presence in AI answers. We compared the main options in Profound vs Bluefish: which tool to choose for AI visibility. Understanding your current score is the starting point for judging whether your bot rule changes are helping or hurting your visibility.
So, is it worth building an AI bot policy for your company?
It's worth it if your site generates significant organic traffic, you want to appear in AI answers, or you have original content to protect. In these cases, a structured bot policy is already urgent.
Somewhere in between: you're an SME with a corporate site and limited traffic. Start with log analysis and a review of your robots.txt: you don't need an enterprise WAF right away, but ignoring the topic is no longer an option.
Not yet if your site is a static showcase with no frequent updates and no organic traffic goals. In this case, the risk is low, but keep an eye on it because the market changes fast.
How we see it at SHM
For us, managing AI bots is one of those cases where technology needs to be understood before it's used, or blocked. We believe AI should be integrated intelligently, not endured or rejected out of fear. Blocking everything without analyzing is the easiest answer, but almost always the wrong one.
We've worked on sites with complex technical catalogs and international traffic, where the distinction between training bots and retrieval bots had a direct impact on visibility in foreign markets. In those cases, the first thing we did was analyze the server logs to understand who was really visiting the site, and why. Only then did we write the rules.
If you want to understand how your site ranks today in AI answers and how to optimize exposure to the right crawlers, the starting point is an SEO strategy for AI that accounts for training, retrieval and agents. If you'd rather automate bot monitoring and response, we can explore a strategic AI integration solution for your technology stack together. And if you want to start from your traffic data, our data processing and business intelligence service is the first concrete step.

Frequently Asked Questions
It depends on your goal. GPTBot is used both for model training and for retrieval in ChatGPT's answers. If you block it completely, you may disappear from ChatGPT's answers to users. The best approach is to analyze your server logs first to understand how GPTBot visits your site, then decide whether to block only training or leave retrieval open.
No. robots.txt is a voluntary agreement: well-behaved bots follow it, but in Q4 2025 about 30% of AI crawlers ignored it. Real protection also requires a CDN and a web application firewall (WAF) configured consistently with your robots.txt rules.
The most reliable way is to analyze your server logs, which record every request along with its user-agent. Many CDNs, like Cloudflare, offer dashboards that show bot traffic in real time. A monthly log review is enough for most sites.
Autonomous agents are bots that browse your site the way a real user would: they click, fill out forms, compare products. Unlike traditional crawlers, they can trigger analytics tracking and are hard to tell apart from human visitors. Blocking them may mean not showing up in the automated evaluations that potential customers delegate to AI.
At least four: the SEO team (who understands the impact on visibility), the IT or DevOps team (who manages the CDN and WAF), the legal team (who assesses copyright and contract risks) and the sales team (who knows which content has strategic value). A policy decided by the SEO team alone, or by IT alone, rarely works.
At least every quarter, and after every significant update from the major AI engines. New bots appear every month and the purposes of existing ones change. A policy written six months ago may already miss bots that visit your site every day.
The question isn't “block or don't block?” — it's “do I really know who is visiting my site and why?”
Related articles
Discover more articles exploring similar topics, selected to offer you a more complete and stimulating perspective. Each piece of content is carefully chosen to enrich your experience.