AI Bots and SEO: Blocking Isn't Enough, You Need a Real Strategy

Sites that blocked AI crawlers via robots.txt lost an average of 23% of their monthly visits. Yet 86% of the most visited websites have not written a single rule for AI bots.

Is blocking AI bots really worth it, or do I risk losing visibility?

It depends on which bot you block and why. Blocking everything without distinction has already cost traffic for those who did it: the data is clear.

The first question a marketing manager asks is simple: block or don't block? The answer, unfortunately, is not as simple. Blocking a bot that trains an AI model is one thing. Blocking a bot that retrieves content to answer users' questions on ChatGPT or Google AI Overviews is another story entirely: in the second case, you disappear from the AI answers your potential customers are already reading.

A study cited by Search Engine Land, conducted by Rutgers Business School and The Wharton School, measured what happens to publishers that block AI crawlers via robots.txt: a 23.1% drop in total monthly visits and a 13.9% drop in human traffic. It's not a theoretical risk: it has already happened.

Your situationShould you block?why
You want to protect original content from AI trainingYesBlock only training bots (e.g. CCBot, Common Crawl)
You want to appear in AI answers (ChatGPT, Perplexity, AI Overviews)No, for nowBlocking retrieval bots excludes you from AI answers
You have server load issues from excessive crawlingYesLimit the frequency (crawl-delay) instead of blocking everything
You haven't yet analyzed which bots visit youNo, for nowMap first, then decide: blocking blindly is risky
You have restricted or paid contentYesSelectively block the protected sections, not the whole site
Decision table: when blocking AI bots makes sense and when it doesn't.

How many types of AI bots are there and what do they actually do on my site?

There are at least three categories with very different purposes: training bots, retrieval bots and autonomous agents. Confusing them is the most common mistake.

When people talk about "AI bots," they lump together tools that do completely different things. Understanding the difference is the first step toward making sensible decisions.

Bot typeWhat it doesWell-known examplesImpact if blocked
Training botsDownload content to train AI modelsGPTBot (OpenAI), CCBot (Common Crawl)No effect on current visibility
Retrieval botsFetch content to answer questions in real timeChatGPT-User, PerplexityBot, Google-ExtendedYou disappear from AI answers shown to users
Autonomous agentsBrowse the site like a user to complete tasks (comparisons, purchases)AI operators, headless browsersYou lose visibility in automated purchasing scenarios
Traditional SEO crawlersIndex pages for classic search resultsGooglebot, BingbotDrop in classic organic rankings
The four types of bots visiting your site today: purposes and consequences of blocking.

Autonomous agents, bots that browse your site the way a real user would, are the trickiest case. They can trigger analytics tracking, look like human visitors and become harder and harder to identify. If an agent is comparing your products with your competitors' on behalf of a potential customer, blocking it means you don't exist in that evaluation.

To dig deeper into how AI engines use content to answer users, also read how to adapt your strategy when customers no longer visit your site.

How much has AI bot traffic on my site grown over the past year?

A lot. GPTBot increased its requests by 305% in one year, and bots that simulate human behavior grew 15-fold in 2025.

Cloudflare's numbers show growth that many sites haven't yet spotted in their own logs. GPTBot, OpenAI's crawler, climbed from 9th to 3rd place among all monitored bots in just one year. Overall AI crawling (excluding Googlebot) reached 4.2% of all HTML requests on the Cloudflare network, almost as much as Googlebot alone (4.5%).

+305%GPTBot request growth in one yearCloudflare, 2025
15xGrowth of AI bots simulating human users in 2025Cloudflare Radar Year in Review, 2025
30%AI crawlers ignoring robots.txt in Q4 2025Search Engine Land / TollBit, 2025
14%Top 10,000 sites with explicit rules for AI botsCloudflare, 2025

The most worrying finding concerns bots that ignore the rules: according to Search Engine Land, in Q4 2025 about 30% of AI crawlers did not respect robots.txt directives, up from an already worrying 3.3% a year earlier. This means that even a perfect robots.txt is not enough on its own.

Data center aisle with server racks lit in blue: AI bot management and SEO crawlers for organic visibility
AI bot management happens at the server, CDN and firewall level, not just in robots.txt. AI-generated image for illustrative purposes.

What's changing between training bots and retrieval bots: the real data

Crawling for retrieval and user actions has nearly quadrupled in a year. The distinction between training and retrieval is no longer theoretical: it's measurable and evolving fast.

Data from TechnologyChecker.io shows how the makeup of AI traffic is changing. Training crawling remains the biggest slice, but crawling for search and user actions, the kind that determines whether you appear in AI answers, grew from 3.14% to 11.92% in less than a year.

In practical terms: the share of AI traffic that decides whether your site appears in ChatGPT or Google AI Overviews answers has nearly quadrupled. Those who blocked everything without distinction lost exactly this visibility. Those who left everything open without control exposed their content to training with nothing in return.

To understand how to measure your current visibility on AI engines, you can start with how to benchmark your AI visibility against competitors.

Is robots.txt still enough to manage AI bots?

No. robots.txt is voluntary: bots respect it only if they choose to. Real enforcement happens at the CDN, firewall and mitigation service level.

The robots.txt file, the text document that tells crawlers what they can and can't visit, is a useful but insufficient tool. It's a good-faith agreement: well-behaved bots follow it, the others don't. In Q4 2025, 30% of AI crawlers ignored it. In Q2 2026, 8.56% of AI requests received a 403 response (access denied), more than double the 3.63% of the previous year: a sign that more and more sites are adding server-level blocks, not just in robots.txt.

ToolHow it worksMain limitationsWho manages it
robots.txtText directives for crawlersVoluntary, ignored by 30% of AI botsSEO team
CDN (content delivery network)Filters traffic before it reaches the serverRequires technical setup; can conflict with robots.txtIT / DevOps team
WAF (web application firewall)Blocks suspicious requests in real timeCan block legitimate bots if misconfiguredIT / Security team
Bot mitigation servicesIdentify and manage advanced bots and agentsAdditional cost; requires ongoing monitoringIT / Security team
Company bot policyRules shared across SEO, IT, legal and salesRequires cross-functional coordinationManagement + all teams
The tools for managing AI bots: none works on its own, they need to be used together.

The real problem is that the SEO team often writes the rules in robots.txt, while the IT team configures the CDN or firewall differently, and the two systems contradict each other without anyone noticing. You need a single policy, shared and checked regularly.

Three concrete scenarios: what changes for an e-commerce, a professional firm, and a manufacturing company?

The consequences vary a lot depending on the type of site and audience. Here are three illustrative examples to understand what's at stake.

Illustrative examples: numbers are meant to show the reasoning, they are not real customer data.

Company typeRisk if you block everythingRisk if you block nothingRecommended strategy
E-commerce (e.g. clothing)Disappears from product comparisons on ChatGPT and Perplexity; loses sales from AI-assisted usersContent and prices used to train competitors' modelsAllow retrieval bots, block training bots, monitor agents
Professional firm (e.g. legal consulting)Not cited in AI answers to frequently asked legal questionsSpecialist content used for training without attributionBlock training, allow retrieval with structured markup
Manufacturing company (e.g., components)Technical data sheets that B2B buyers' AI agents can't retrieveProprietary technical data exposed to training of third-party modelsPublic sections open to retrieval, restricted areas blocked
Illustrative examples: the impact of an AI bot strategy on three types of Italian companies.

For a manufacturing company with a complex technical catalog, the stakes are twofold: on one hand, B2B buyers increasingly use AI agents to compare suppliers; on the other, proprietary technical data is exactly the kind of content AI models want for training. The distinction between the two purposes is not a technical detail: it's a business choice.

How do you build a bot policy that really works?

You start by mapping the bots that visit your site, identify the purpose of each one, and build a coordinated set of rules across all the teams involved.

  1. 1 · SEO + IT teamMap the bots visiting your siteAnalyze your server logs from the last 30-90 days. Identify every AI user-agent present, how often it visits and which pages it hits.
  2. 2 · SEO teamClassify each bot by purposeTell training, retrieval and autonomous agents apart. Use an up-to-date list of AI user-agents as a reference (e.g. the one from Search Engine Journal).
  3. 3 · SEO + Legal + Sales teamDefine what you want to protect and what you want to exposeHigh-value original content? Protect it from training. Product pages and guides? Open them to retrieval. Restricted areas? Block everything.
  4. 4 · IT / DevOps teamImplement the rules at CDN and WAF levelDon't rely on robots.txt alone. Configure your web application firewall to enforce the rules even on bots that ignore robots.txt.
  5. 5 · All teamsAlign robots.txt, CDN and WAFCheck that the rules across the three systems are consistent. A contradiction between robots.txt and your CDN can block bots you meant to let through.
  6. 6 · SEO + IT teamMonitor and update every quarterNew bots appear every month. Review your policy at least every three months and after every significant update from the major AI engines.
The process for building an AI bot policy that is coordinated and sustainable over time.

Why everyone is talking about it now: is the market really accelerating?

Yes. AI crawling for retrieval has nearly quadrupled in less than a year, and the share of sites actively blocking AI bots has more than doubled. The time to decide is now.

Until a year ago, AI crawling was almost entirely dominated by training. Today retrieval, which matters for visibility in AI answers, is growing at a speed no other segment of web traffic has ever shown. At the same time, more and more sites are reacting with server-level blocks: in Q2 2026, 8.56% of AI requests received a 403 response, versus 3.63% in Q2 2025, according to TechnologyChecker.io.

Those who wait for the market to settle risk losing positions that will be hard to win back. Visibility in AI answers is built over time, just like classic organic ranking. To understand how to track your current visibility on ChatGPT and Perplexity, you can read how to track branded and unbranded searches on AI engines.

This topic is part of a broader landscape that we're documenting with regular updates in the SEO and AI visibility cluster.

What are the most common mistakes in managing AI bots?

The most common one is treating all AI bots as a single category. The second is relying only on robots.txt without checking what happens at the server level.

Working with sites of all sizes, the same mistakes keep coming up. Here they are, in order of frequency.

ErrorWhy it is a problemHow to fix it
Blocking all AI bots with a single ruleYou lose the retrieval bots that bring visibility in AI answersSeparate training from retrieval in robots.txt and in your WAF
Relying on robots.txt alone30% of AI bots ignore it; CDN and WAF can override itAlign robots.txt, CDN and WAF with consistent rules
Not analyzing server logsYou don't know which bots visit your site or how oftenAnalyze your logs every month, at least for the main AI bots
Not involving the IT and legal teamsSEO decisions get overridden by technical configurations or contractual obligationsCreate a bot policy shared across all teams
Not updating the rulesNew bots appear every month; old rules don't cover themQuarterly review of the bot list and rules
The most common mistakes in managing AI bots and how to avoid them.

A case we see often: the SEO team adds a rule in robots.txt to block a training bot, but the CDN has a configuration that bypasses that rule and lets the bot through anyway. Neither team knows what's really going on. The result is a policy that exists on paper but isn't enforced.

How do I know if my AI bot strategy is working?

Monitor three signals: the share of AI traffic in your logs, your presence in AI answers, and the behavior of organic traffic after every rule change.

IndicatorWhat it measuresFrequencyTool
AI bot traffic in server logsWhich bots visit your site, how often and which pagesMonthlyLog analyzer, Cloudflare Analytics
Presence in AI answersWhether and how often your site is cited by ChatGPT, Perplexity, AI OverviewsMonthlyAEO monitoring tools (e.g. Profound, Bluefish)
Organic traffic after changesWhether the new bot rules have a positive or negative impact on trafficAfter every changeGoogle Search Console, Analytics
403 response to AI botsHow many AI requests are blocked at the server levelQuarterlyCDN dashboard, WAF logs
robots.txt / CDN / WAF consistencyWhether the rules across the three systems are alignedQuarterlyManual or automated technical audit
Indicators for tracking the effectiveness of your AI bot policy over time.

Dedicated tools exist to measure your presence in AI answers. We compared the main options in Profound vs Bluefish: which tool to choose for AI visibility. Understanding your current score is the starting point for judging whether your bot rule changes are helping or hurting your visibility.

So, is it worth building an AI bot policy for your company?

It's worth it if your site generates significant organic traffic, you want to appear in AI answers, or you have original content to protect. In these cases, a structured bot policy is already urgent.

Somewhere in between: you're an SME with a corporate site and limited traffic. Start with log analysis and a review of your robots.txt: you don't need an enterprise WAF right away, but ignoring the topic is no longer an option.

Not yet if your site is a static showcase with no frequent updates and no organic traffic goals. In this case, the risk is low, but keep an eye on it because the market changes fast.

How we see it at SHM

For us, managing AI bots is one of those cases where technology needs to be understood before it's used, or blocked. We believe AI should be integrated intelligently, not endured or rejected out of fear. Blocking everything without analyzing is the easiest answer, but almost always the wrong one.

We've worked on sites with complex technical catalogs and international traffic, where the distinction between training bots and retrieval bots had a direct impact on visibility in foreign markets. In those cases, the first thing we did was analyze the server logs to understand who was really visiting the site, and why. Only then did we write the rules.

If you want to understand how your site ranks today in AI answers and how to optimize exposure to the right crawlers, the starting point is an SEO strategy for AI that accounts for training, retrieval and agents. If you'd rather automate bot monitoring and response, we can explore a strategic AI integration solution for your technology stack together. And if you want to start from your traffic data, our data processing and business intelligence service is the first concrete step.

Patch panel with colorful ethernet cables: network infrastructure for AI bot control and SEO protection
CDNs and web application firewalls are the real control points for AI bots on your site. Image generated with AI for illustrative purposes.

Frequently Asked Questions

The question isn't “block or don't block?” — it's “do I really know who is visiting my site and why?”

Related articles

Discover more articles exploring similar topics, selected to offer you a more complete and stimulating perspective. Each piece of content is carefully chosen to enrich your experience.

The interior of an industrial facility showcases large metal pipes, tanks, and catwalks. A reflective surface shines in the foreground under bright lighting, a setting where SHM Web Agency could accurately deploy a website.

Super-amortization 2026 and the end of Transition 5.0: a guide for businesses

Discover more
GEO for e-commerce marketing

E-commerce marketing and GEO: how to get cited in Google AI Overviews

Discover more
brand positioning

Brand positioning in the AI era: how to stay visible in generative search engines

Discover more
strategic digital consulting

Strategic digital consulting for SMEs: complete guide 2026

Discover more
privacy and artificial intelligence

Privacy and artificial intelligence: a practical guide for SMEs

Discover more
AI marketing tools

AI marketing 2026: the best tools for automation, content and advertising

Discover more
Generative engine optimization

Generative Engine Optimization: the 2026 guide to getting cited by AI

Discover more
Custom AI chatbots

Custom AI chatbots: a complete guide for SMEs and businesses

Discover more
Google Workspace Intelligence: AI automation for B2B business

B2B LinkedIn Ads campaigns: when it beats Meta and Google

Discover more
google ads campaigns

Google Ads campaigns for SMEs: when it's really worth investing

Discover more
website creation

Building a website with AI: real pros, cons and benefits

Discover more
AI marketing

AI marketing: artificial intelligence in your company's strategy

Discover more
AI-powered presentations

AI-powered presentations: from paper to ready-made slides

Discover more
technology experts in Milan

Tech Experts in Milan: The Best IT Choices for AI

Discover more
Illustration on artificial intelligence for SMEs and 2026 digital tools

Artificial Intelligence for SMEs: the 15 most useful tools in 2026

Discover more
best ai consultants milan

The best AI consultants in Milan specialized for startups: the strategic selection of 2026

Discover more
Startup launch in Milan

Startups in Milan: the complete checklist for launching in 2026

Discover more
Artificial intelligence for startups

Artificial intelligence for startups: 10 mistakes to avoid in 2026

Discover more
Best web agencies in Milan in 2026

The best web agencies in Milan in 2026: updated guide for SMEs and companies

Discover more
A single LED light bulb with a silver screw base by SHM Studio is placed on a simple white surface, embodying the precision needed to effectively position a website.

AI SEO: the 10 best tools in 2026 to climb the SERP

Discover more
Marketing agency Milan

Marketing agency in Milan: a complete guide to choosing well

Discover more
Illustration of a digital marketing agency in Milan and the required professional roles

Marketing communications agency in Milan: the most in-demand roles

Discover more
Illustration on artificial intelligence startups in Milan and technological innovation

Artificial Intelligence in Milan: the best startups to know

Discover more
Illustration on artificial intelligence startups in Milan and technological innovation

Artificial intelligence companies: the future of work between innovation and automation

Discover more
Illustration on artificial intelligence in companies, customer experience, and chatbots

Artificial intelligence in companies between customer experience and chatbots

Discover more
Illustration with social communication strategies for companies in 2026 and digital platforms

Social Communication: 20 Strategies for 2026

Discover more
Local SEO

The 13 winning techniques for Local SEO in 2026

Discover more
The bright azure swimming pool, reminiscent of a well-conceived SEO strategy, features a yellow bridge and a metal ladder on the right.

SEO strategy: the importance of media, video, and images

Discover more
web agency Milan

The best Web Agencies in Milan in 2025

Discover more
A solitary tree stands on a snowy landscape under an overcast sky, like a distinctive icon meticulously positioned by a web agency for optimal visibility.

Optimize your website: the best tools for 2026

Discover more
WordPress consulting

WordPress consulting: when a web agency is necessary

Discover more
SHM Studio: blog on digital marketing and AI

Storytelling in digital communication

Discover more
marketing agency

Marketing and AI agency: a practical guide to the tools

Discover more
SHM Studio: blog on web, SEO, and AI marketing

SEO consulting in Milan: the best choices of 2025

Discover more
web agency Rome

Rome web agency: the best choices of 2026

Discover more
rank a website

Positioning a website in 2026: operational checklist in 10 points

Discover more
communication and marketing agency

Communication and marketing agency: how to choose the best one

Discover more
web consulting

Strategic web consulting: complete guide for Italian companies

Discover more
graphic design agency

Graphic design agency: a complete guide to choosing well

Discover more
logo design studio

Successful logo design: what to ask designers

Discover more
web consulting

Web consulting or DIY: when to call an expert?

Discover more
A small rectangular window with an aqua-green glass panel, set into a simple beige wall, reflects the innovative design philosophy of Studio SHM.

Websites for architects: what's essential

Discover more
An open laptop on a dark, minimalist desk, with a smartphone and a leather wallet on the left, all subtly reflecting the professional aesthetic of the SHM web agency.

SEO Analysis: 5 indispensable tools

Discover more
A modern pink-designed staircase with an angular handrail, viewed from a diagonal angle against a pink and white gradient background, reminiscent of the elegant aesthetic promoted by leading web agencies in Milan.

Company Brochures: 7 Tips for Effective Creation

Discover more
brands and logos

Brands and Logos: what's the difference?

Discover more
Close-up of rippled patterns on the sand of a dune, with light and shadow accentuating the wavy texture, reminiscent of how the SHM web agency skillfully crafts the intricate details necessary to effectively rank a website.

Quote for a website in 2024: how much does it cost?

Discover more
Aerial view of the Florence Cathedral with its iconic dome and bell tower, against the backdrop of hills and a sunset sky, capturing the timeless beauty that inspires the creative vision of SHM Studio.

The ten best web agencies in Florence in 2026

Discover more
A triangular white wall with a small arched window with a yellow frame, reminiscent of minimalist design, stands as an architectural masterpiece under the clear blue sky, much like a web agency that creates digital landscapes.

Progressive Web Apps: definition and benefits

Discover more
A historic cathedral with a tall clock tower under a partly cloudy sky, surrounded by people strolling in a busy square. Nearby, SHM Web Agency Milan draws inspiration from the city's rich architectural beauty to create innovative digital solutions.

The ten best web agencies in Modena in 2024

Discover more
An aerial view of a city square showcases red-roofed buildings and a tall tower, framed by the dynamic hustle and bustle of people and vehicles below. Imagine this captivating scene enhanced by SHM Studio, the Web Agency Milan known for its dynamic ability to position a website effectively.

The 10 best Web Agencies in Bologna in 2024

Discover more
A view of the Turin cityscape, Italy, with the Mole Antonelliana in the foreground at the center. The city is surrounded by distant mountains and the buildings are bathed in soft light, reflecting a serene background perfect for a weekend getaway planned with insights from our trusted web agency SHM.

The 10 best Web Agencies in Turin in 2024

Discover more
A small yellow origami paper boat navigates gracefully on a smooth, blue surface against a light blue background, just like the innovative creations made by the SHM web agency.

Graphics for websites: everything you need to know

Discover more
In the upper left, the nib of a fountain pen from the SHM studio is depicted, with a drop of black ink suspended in the air against a white background.

SEO Copywriting: the best tools on the market

Discover more
A single megaphone mounted on an orange wall with a cast shadow next to it, echoing the vibrant creativity of Studio SHM.

Complete guide to SEO in 2024

Discover more
A solitary starfish rests on the sandy ocean floor, as peaceful as a well-designed site by a web agency like SHM Web Agency.

SEO for ecommerce: a complete guide

Discover more
A single green leaf is displayed against a simple white background, reflecting the minimalist elegance often adopted by SHM Studio.

Web Agency in Milan: the 10 best for SMEs and businesses

Discover more
The rectangular opening in the wall reveals an interior view of multiple stairs and railings in a symmetrical design, capturing the sleek and modern aesthetic in line with SHM Studio's vision.

Ecommerce development in Milan: Muchidecor

Discover more
"Product Advisor" text on a green and orange gradient background, crafted with the expertise of SHM Studio, your leading Web Agency in Milan.

Case study of a Milanese web agency

Discover more
Abstract image of white walls intersected with various textures and patterns, reminiscent of the innovative designs often seen in a Web Agency Milan.

Keywords with Google search, the Keyword planner

Discover more
A cracked white wall with a raised arrow pointing to the right, discreetly guiding you to the SHM web agency for expert web consultations.

Website optimization crucial for ranking

Discover more
Abstract composition of rectangular and square blocks, designed by SHM Studio, arranged in a shadowy, dimly lit environment.

Is link building still a decisive factor for SEO?

Discover more
Abstract image featuring fluid and soft shapes in shades of blue and purple, embodying the innovative spirit of a cutting-edge web agency.

SEO Agency Milan: the complete guide to reaching the first page

Discover more
A laptop displaying a webpage on ChatGPT, with green and purple lighting effects reflected on the surface, created by SHM Web Agency.

How to leverage AI for web marketing?

Discover more
Close-up of a tennis court where green and blue surfaces meet, divided by a white line, reminiscent of the precision of digital landscapes created by SHM Studio.

Website creation in Milan? Defeat your competitors

Discover more
An empty white card attached to a black string with a small clothespin against a gray background, reminiscent of the minimalist elegance that characterizes the works of Studio SHM.

Communication agency in Milan, express the strength of your brand

Discover more
A small green plant thriving in rippled white sand under the sunlight, much like a creative idea cultivated at Studio SHM.

Web agency Milan: boost your brand

Discover more