- Is it really worth creating separate content for AI bots?
- What's really happening: what is the AI "shadow web"?
- What counts as shadow web and what doesn't
- How active are AI bots on your site right now?
- What Google says: is cloaking toward AI bots already banned?
- What happens in practice if AI cites different versions of your brand?
- Three real-world scenarios: what does your company risk?
- What tools or approaches exist to optimize for AI without taking risks?
- Why it's in the spotlight now: is the AI market really fragmented?
- How to audit the content exposed to AI bots: the concrete steps
- What mistakes are most common when optimizing for AI?
- How do I know if my AI-SEO strategy is working without creating risks?
- So, is it worth creating separate content for AI bots at your company?
- How we see it at SHM
- Frequently Asked Questions
- Creating separate content only for AI bots (not visible to users) is comparable to cloaking, a practice Google has already penalized, and since May 2026 that includes AI Overviews and AI Mode.
- AI bots are growing fast: according to Cloudflare, OpenAI's GPTBot grew its share of all crawlers from 5% to 30% in one year.
- The solution isn't to block bots or build a "shadow web": it's to audit your public content so it's readable by both users and AI, without duplicating or hiding information.
- The real risk isn't just a penalty: it's that AI cites information that differs from what's on your site, creating contradictions that damage customer trust.
Are you optimizing your content for AI bots with files and pages your customers never see? That shortcut has a precise name in SEO — and Google has already said it will penalize it.
Is it really worth creating separate content for AI bots?
No, if that content isn't also accessible to regular users. The logic seems sound, but the legal and reputational risk outweighs the tactical advantage.
Many companies are discovering that AI systems — ChatGPT, Gemini, Perplexity — can't easily find the information they need on their site. The instinctive response is to create dedicated files, pages in Markdown format (a simplified text format that bots can read) or separate endpoints (web addresses meant only for machines) with more detailed or structured information.
There's just one problem: if that version says something different from what your customers read, you've created a shadow web. And search engines — Google first among them — have already said they treat it as spam.
| Your situation | Is it worth it? | why |
|---|---|---|
| You make the same content readable in multiple formats (HTML + Markdown) | Yes | A legitimate practice, similar to XML sitemaps. No risk. |
| You add structured data (schema markup) to existing pages | Yes | It helps both traditional search engines and AI understand context. |
| You create an llms.txt file with links to the site's public pages | Yes | Acceptable if it points to content already visible to users. |
| You publish information for bots only, not accessible to users | No, for now | Comparable to cloaking: risk of a direct penalty. |
| You describe the brand differently to bots than on the public site | No, for now | AI may cite contradictory versions, damaging trust. |
What's really happening: what is the AI "shadow web"?
It's a body of content created specifically for AI bots, not visible or not easily reachable by human users. The original source of this analysis is Search Engine Journal.
The analysis published on Search Engine Journal points to a real trend: companies struggle to publish content on their official site (slow internal processes, rigid CMSs, endless approvals) and find it easier to create text files or alternative pages to offer to AI bots.
The result is a double track: what the customer sees and what the machine sees. In classic SEO this is called cloaking — showing search engines different content from users — and it has been explicitly prohibited by Google's policies since 2011.
What's new in 2026 is that Google has extended that same rule to generative AI answers, as we'll see in the section on regulations.
What counts as shadow web and what doesn't
- Not shadow web: XML sitemaps, structured data, product feeds, public APIs, robots.txt files, llms.txt pointing to public pages.
- Shadow web: Markdown pages with information not found on the site, endpoints with brand descriptions that differ from the public ones, text files with content created only to influence AI answers.
How active are AI bots on your site right now?
A lot more than you think. OpenAI's GPTBot has multiplied its crawling share sixfold in just one year, and AI traffic to websites has grown sixteenfold from 2024 to today.
Understanding the scale of the phenomenon helps you make the right call. AI bots are no longer a lab curiosity: they are already reading your site, and they're doing it more and more systematically.
Sixteen times in two years: traffic from AI engines went from 0.02% to 0.32% of the global total, according to SE Ranking. In absolute terms it's still a small share, but the speed of growth is what matters if you need to plan today.
According to the Cloudflare Radar Year in Review 2025, AI bots as a whole have already come close to the volume of Googlebot. And among them, OpenAI's GPTBot grew 305% between May 2024 and May 2025, going from a 5% to a 30% share of all the AI and search crawlers observed by Cloudflare, as reported on the Cloudflare blog about 2025 crawlers.
What Google says: is cloaking toward AI bots already banned?
Yes. As of May 2026, Google has officially confirmed that its anti-spam policies also apply to generative AI answers, including cloaking aimed at AI crawlers.
This isn't a future hypothesis: it's already policy. In May 2026, Google updated its Spam Policies to clarify that they also apply to AI Overviews and AI Mode, the generative answer sections in search results. Search Engine Land confirmed it.
Cloaking aimed at AI bots explicitly falls into this category. And the system that detects it, Google SpamBrain, uses multimodal rendering (it compares what the crawler sees with what a regular browser sees) to spot discrepancies, as reported by Search Engine Land in 2025.
| Year | Google update | What changes for companies |
|---|---|---|
| 2011 | Panda | Penalizes low-quality content |
| 2012 | Penguin | Penalizes link spam and link manipulation |
| 2022 | Helpful Content System | Rewards content that's useful to people, not to search engines |
| March 2024 | Scaled Content Abuse + new spam policies | Covers mass production of pages to manipulate rankings, whether AI-made or human-made |
| May 2026 | Spam Policy extended to AI Overviews and AI Mode | Cloaking toward AI crawlers is explicitly banned |
What happens in practice if AI cites different versions of your brand?
The customer gets contradictory information: one version from the AI, another from your site. This erodes trust before they even reach your site.
Imagine a manufacturing company that sells industrial components. On the public site, delivery time is listed as 15 business days. In a file created for AI bots, to look more competitive, it wrote 7 days. A potential customer asks ChatGPT: the answer says 7 days. They land on the site and read 15. The negotiation starts with a crack already in it.
This scenario, presented here as an illustration, is exactly the risk the original source highlights: who decides which version is more reliable? And the answer, in the short term, is nobody. In the medium term, it will be the AI systems that learn to devalue inconsistent sources.
The issue of conflicting brand information in AI Search is already documented: AI tends to favor the sources that are most consistent and verifiable over time. A shadow web works against that logic.
Three real-world scenarios: what does your company risk?
The risk varies depending on how far your AI-specific content diverges from your public content. Here are three typical cases from Italian companies.
| Company type | Current practice | Concrete risk | Recommended action |
|---|---|---|---|
| E-commerce (e.g., auto parts) | llms.txt file with prices and availability that are out of date compared to the live catalog | AI cites wrong prices; customer arrives and finds a discrepancy | Sync llms.txt with the product feed in real time |
| Professional firm (e.g. tax consulting) | Markdown page with services not listed on the official site | Google SpamBrain detects the discrepancy; risk of a penalty | Publish those services on the site before exposing them to bots |
| Manufacturing company (e.g., precision metalwork) | Endpoint with English technical data sheets for GPTBot only | No immediate risk if the sheets are identical to the site; risk if they add claims that can't be verified | Check consistency between endpoints and public product pages |
What tools or approaches exist to optimize for AI without taking risks?
There are legitimate practices and risky ones. The difference always comes down to consistency between what bots see and what users see.
| Tool / Approach | Legitimate? | Operational notes |
|---|---|---|
| Schema markup (structured data) | Yes | Helps AI and search engines understand context without duplicating content |
| llms.txt with links to public pages | Yes | Acceptable if it only points to URLs that are already indexed and consistent |
| Up-to-date XML sitemap | Yes | Essential foundation for any AI visibility strategy |
| Cloudflare Markdown for Agents (same content, different format) | Yes | Legitimate if the content is identical to the original page |
| Markdown pages with extra information not on the site | No, for now | Cloaking risk: avoid |
| Text files with brand descriptions that differ from the site | No, for now | Contradiction that AI can verify; reputational and SEO risk |
| Endpoints with prices or claims not found on the public site | No, for now | Direct violation of Google's 2026 spam policies |
Why it's in the spotlight now: is the AI market really fragmented?
Yes, and fragmentation complicates everything. There isn't a single "AI engine" to optimize for: ChatGPT, Gemini and others use different pipelines and already have very different market shares.
A year ago ChatGPT dominated AI traffic with a 76% global share. Today, according to Similarweb (May 2026), it has dropped to 53%, while Gemini has risen from 9% to around 27-28%. The market is fragmenting fast.
This means optimizing for a single AI system isn't enough. It also means that the rules of trust and verification vary from platform to platform: what works with one system might not work with another, or might even get you penalized. To dive deeper into SEO and AI visibility as a whole, our dedicated cluster collects all the latest analyses.
It's worth noting that 14% of the top thousand sites by global traffic have chosen to block AI bots via robots.txt, according to Cloudflare. It's an understandable defensive choice, but it rules you out of AI visibility entirely, a sacrifice that gets more costly as AI traffic grows.
How to audit the content exposed to AI bots: the concrete steps
The audit starts by mapping everything bots can read on your domain, then checks it for consistency against your public content. Here's how to do it systematically.
- 1 · SEO / Marketing teamMap every access point for botsCollect robots.txt, sitemaps, llms.txt, Markdown files, public API endpoints and product feeds. Everything a bot can reach on your domain.
- 2 · SEO / Marketing teamCompare bot content with public pagesFor each resource you find, check that the information is identical to, or a subset of, what's on the site. Any addition or difference is a risk signal.
- 3 · Legal / Compliance teamCheck sensitive claimsPrices, delivery times, certifications, warranties: if they appear in bot-only resources, assess both the SEO and the contractual risk.
- 4 · SEO / Marketing teamReview crawl logsUse Google Search Console and server logs to see which bots are visiting which URLs. GPTBot, ClaudeBot and PerplexityBot have identifiable user agents.
- 5 · Content / CMS teamPublish on your site what you want AI to citeIf information is useful for AI, it should be useful for users too. Remove the internal barriers to publishing: that's the real problem to solve.
- 6 · SEO / Marketing teamTrack AI citations over timeRegularly check what ChatGPT, Gemini and Perplexity say about your brand. If they cite information that isn't on your site, you have a consistency problem to fix.
What mistakes are most common when optimizing for AI?
The three most common mistakes all stem from the same root: treating AI bots as a separate audience instead of an alternative way to access the same content.
- Creating 'AI-only' content without publishing it on your site. This is the most direct form of shadow web. The fix is simple: if it's worth an AI knowing it, it's worth your customer knowing it too.
- Using llms.txt as an editorial shortcut. The llms.txt file (a proposed standard for guiding language models to a site's content) is useful if it points to pages that already exist. It becomes a problem if it replaces publishing content on your site.
- Ignoring consistency between product feeds and your site. For e-commerce, prices and availability in feeds must be synced in real time. An outdated feed read by an AI bot leads to wrong answers for customers.
- Blocking all AI bots out of fear. 14% of top sites do it, but it's a drastic solution. It's better to manage permissions selectively in robots.txt, letting trusted bots access public content.
- Not monitoring what AI says about your brand. As explained in the article on the AI visibility score and its two dimensions, citations and mentions are different metrics and should be tracked separately.
How do I know if my AI-SEO strategy is working without creating risks?
There are specific indicators that signal both AI visibility and content consistency. Monitoring them together is the only way to grow without risking penalties.
| Indicator | How it is measured | Positive signal | Risk signal |
|---|---|---|---|
| Brand AI citations | Manual queries on ChatGPT, Gemini, Perplexity | Brand cited with information consistent with the site | Information that differs from what's on the site |
| AI bot traffic | Server logs + Google Search Console | Gradual growth of GPTBot, ClaudeBot, PerplexityBot | Bots blocked unintentionally or missing |
| Bot/site content consistency | Manual audit or crawling tool | No discrepancies between bot resources and public pages | Information found only in bot-only resources |
| Indexing coverage | Google Search Console > Coverage | Key pages indexed and error-free | Important pages excluded or with crawl errors |
| Organic rankings | SEO tools (e.g. SE Ranking, Semrush) | Stability or growth after Google updates | Sudden drops after spam policy updates |
To build a systematic process, the content validation workflow for AI Search we've analyzed offers a concrete starting point for SMEs.
So, is it worth creating separate content for AI bots at your company?
It's worth it if you already have a site with complete, up-to-date content and just want to make it more readable for bots (structured data, sitemaps, an llms.txt that points to existing pages). In this case, optimizing the format is legitimate and useful.
A middle ground: your site has information gaps you'd like to fill "just for AI." The right way is to publish that content on your site, not to create parallel resources. It takes more work, but it's the only sustainable solution.
Not yet if you're considering creating pages, files or endpoints with different or additional information compared to your public site, thinking bots aren't checked. Google SpamBrain already compares the two versions, and the 2026 policies are explicit.
How we see it at SHM
For us, AI should be integrated to free companies from routine, not to create shortcuts that backfire later. The shadow web is exactly that: a shortcut that sidesteps the real problem (publishing useful content on your site) and creates a new one (inconsistency that search engines and AI can verify).
We work with manufacturing and B2B companies that have complex technical catalogs, slow approval processes and rigid CMSs. We know the frustration of being unable to publish what customers, and now AI too, are looking for. The answer isn't to bypass the site: it's to simplify the process that gets content onto the site.
In practice, this means starting with a technical SEO audit that maps everything bots see today, identifying the discrepancies, and then building an editorial plan that fixes the gaps at the root. It's the same approach we use for SEO for AI and for the strategic integration of AI into our clients' processes.
If you want to understand what bots see on your site today, and whether anything risks working against you, the starting point is a no-obligation conversation. You can also explore the topic of brand context in AI Search to understand how machines build their image of your brand.
Frequently Asked Questions
Cloaking is the practice of showing different content to search engines (or bots) than what regular users see. Google bans it because it distorts a site's relevance and authority, undermining the quality of results. Since May 2026, this rule explicitly applies to generative AI answers like AI Overviews and AI Mode as well.
No, if it points to pages on your site that are already public and indexed. The llms.txt file is a proposed standard for guiding language models to the most useful content on a site: it's legitimate when used as an index of existing resources. It becomes problematic if it contains information that isn't found anywhere else on the site or describes the brand differently from the public pages.
You can check your site's server logs and filter by known user agents: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Googlebot-Extended. Google Search Console also shows Googlebot's crawl stats. For a more complete analysis, tools like Cloudflare or dedicated log analysis solutions let you see bot traffic in detail.
Yes, you can selectively block AI bots in your robots.txt file without touching Googlebot. 14% of the world's top 1,000 sites already do. But the choice has a cost: it excludes your site from citations by ChatGPT, Gemini and other AI systems, a loss that matters more and more as AI traffic grows.
The first step is to verify the source: AI tends to cite what it finds on the web, including bot-specific resources you may have created yourself. If the wrong information comes from external resources, the strategy is to strengthen the authoritative content on your site and build consistency across all the sources that talk about your brand. The article on conflicting information in AI Search goes deeper into this topic.
For a mid-sized site (100-500 pages), a basic audit takes 2-4 days of work: mapping the resources accessible to bots, comparing them with public content, and analyzing crawl logs. For sites with complex technical catalogs or product feeds, it takes longer. What matters is doing it before you get a penalty, not after.
The shadow web isn't a technical problem: it's the consequence of not solving the real problem, which is making useful content accessible on your site.
Related articles
Discover more articles exploring similar topics, selected to offer you a more complete and stimulating perspective. Each piece of content is carefully chosen to enrich your experience.
