Sitemaps and RSS for AI crawlers: what to do now to get found

Google's John Mueller has seen AI crawlers access his sitemap and RSS feed in his own server logs. If you haven't optimized these files yet, you're losing visibility on Perplexity, ChatGPT and Gemini every day.

Is it really worth optimizing sitemaps and RSS for AI crawlers?

Yes, and the reason is simple: AI crawlers have no portal where you can submit your sitemap, so they only find your content if the file has a standard name or if a reachable RSS feed exists. It costs little and opens up a visibility channel across all the major artificial intelligence systems.

AI crawlers, the automated robots that collect content to train or feed systems like ChatGPT, Perplexity and Gemini, work differently from Googlebot. Google gives you Search Console (the control panel for webmasters), where you can submit your sitemap. AI crawlers have nothing like it.

This means that if you want your content to be found and potentially cited by these systems, you need to make your sitemap and RSS feed easy to discover on their own. That's exactly what Google's John Mueller confirmed on the Search Off the Record podcast on October 1, 2025.

What exactly did Mueller say, and what does it mean for you?

Mueller confirmed that he has seen AI crawlers access both his sitemap and his RSS feed in his own server logs. He pointed to two practical options: use the standard file name sitemap.xml or enable an RSS feed.

The key point is this: AI crawlers look for content on their own, without waiting for you to invite them. They check the robots.txt file (the document that tells robots what they can and can't do on your site), look for files with standard names like sitemap.xml, and follow the links to RSS feeds they find in the HTML head of your pages.

Mueller also clarified what happens if you want to keep your sitemap private: you can give it an unusual name and leave it out of robots.txt, submitting it only to Google. The flip side is that no other system, AI crawlers included, will ever find it.

On the subject of server logs: they're the files that record every visit to your site. Analyzing them tells you which bots visit you, when and how often. It's a free tool that often gets overlooked, but it's valuable for understanding who is already reading your content.

How much traffic are AI crawlers already sending to your site?

More than you think. OpenAI's GPTBot has grown 305% in one year and now reaches almost 29% of sites on the Cloudflare network. AI traffic is already in your logs, even if you're not measuring it.

Data published by Cloudflare in July 2025 shows explosive growth in AI crawler traffic. Between May 2024 and May 2025, total AI and search crawler traffic grew 18%, but GPTBot alone recorded a 305% increase.

This means OpenAI's robots now visit many more sites than they did a year ago. If your site isn't set up to make this discovery easy, you're ceding ground to those who already have.

Looking at the makeup of traffic, AI bots (excluding Googlebot) accounted for an average of 4.2% of all HTML requests on the Cloudflare network in 2025. Googlebot alone generated 4.5%, more than all the other AI crawlers combined. But the gap is closing fast, according to the Cloudflare Radar Year in Review 2025.

Who are the main AI crawlers visiting websites today?

OpenAI's GPTBot leads the ranking with 30% of AI traffic, followed by Anthropic's ClaudeBot at 21% and Meta-ExternalAgent at 19%. The landscape has changed dramatically in just one year.

A year ago, Bytespider (the crawler from ByteDance, TikTok's parent company) dominated with 42% of AI traffic. By May 2025 it had dropped to 7.2%, while GPTBot took the lead. This rapid shift confirms that the AI crawler landscape is still evolving, and it's worth monitoring in your server logs.

Almost all of this traffic, around 80%, is devoted to model training, meaning the collection of data used to train AI systems. Only about 18% serves search indexing. This explains why being present in training data can influence the answers these systems give in the future on topics in your industry.

To dig deeper into how this affects traditional SEO, also read what to do for SEO when Google AI Mode reduces clicks.

What changes in practice: who does what on your team?

Optimizing your sitemap and RSS is a quick technical task, but it requires coordination between whoever manages the site and whoever produces the content. Here's how the work is split.

Three concrete scenarios: how does the situation change for an Italian company?

The impact depends on the type of site and how often you publish. Here are three illustrative cases to show what to expect.

Illustrative examples: the names and numbers are meant to show the reasoning and are not data from real clients.

Is llms.txt really an alternative to the sitemap?

No, at least for now. Mueller was clear: no AI system currently uses llms.txt as a replacement for the sitemap. It's a Markdown file (a simple text format) that lacks the rigid structure needed to be processed like a sitemap.

The llms.txt file is a proposal born in 2024 to give AI systems a simplified version of a site's content. The idea is interesting, but Mueller said explicitly: "The hope is bigger than the reality. Right now none of this is happening.".

In August 2025 he had already observed that the only crawlers claiming to accept Markdown files were SEO tools, not real AI systems. Google itself, in its guide to optimizing for generative AI, listed llms.txt among the tactics that sites don't necessarily need to implement.

The practical advice: you can add llms.txt to your site with no problem, but it doesn't replace sitemap.xml and RSS. Treat it as an optional experiment, not a priority.

Why are people talking about this now and not a year from now?

Because AI crawler traffic is already significant and growing fast, and those who optimize today have an edge over those who wait. The market is also becoming polarized: some publishers block crawlers, others welcome them.

In July 2025, Cloudflare enabled default blocking of all major AI crawlers for every new domain registered on its network, which covers about 22.4% of the global public web, according to Search Engine Land. This means a growing part of the web is becoming invisible to AI crawlers.

If you want to be found, the time is now: less competition in training data means a better chance of being cited. The management of product feeds and the Business Profile follows the same logic: whoever structures their data correctly today reaps the rewards tomorrow.

The drop in GPTBot's coverage doesn't mean the crawler is less active: it means more sites are blocking it. If you don't block it and have a well-configured sitemap, you become proportionally more visible. The topic ties into indexing speed and disappearing pages: content quality remains the deciding factor for AI crawlers too.

On the RSS front, BuiltWith data (July 2025) shows that 35% of the world's 10,000 most-visited sites use RSS. In Italy, over 811,000 sites with RSS feeds are active: the technology is mature and widespread, not a novelty to adopt with caution.

How to set it all up: concrete steps to start today

Three technical steps are enough, all doable in under an hour on any modern CMS. Here's the right sequence.

1. Check your sitemap's name. Log in to your CMS dashboard (WordPress, Shopify, etc.) and check what the sitemap file is called. If it isn't sitemap.xml, consider renaming it or adding an alias.
2. Add the Sitemap line to robots.txt. Open your site's robots.txt file (usually available at tuosito.it/robots.txt) and add this at the bottom: Sitemap: https://tuosito.it/sitemap.xml. This line is read by all crawlers that support the sitemaps protocol, regardless of your other rules.
3. Enable and check the RSS feed. On WordPress it's active by default at tuosito.it/feed. Make sure it's linked in the HTML head with a link rel="alternate" type="application/rss+xml" tag. If it's missing, add it or ask your developer.
4. Analyze your server logs. Ask your hosting provider for access to the log files (access logs). Search for the strings GPTBot, ClaudeBot, anthropic-ai, meta-externalagent to see who is already visiting you and how often.
5. Decide whether to block or welcome them. If you want to keep AI crawlers out, add the lines User-agent: GPTBot and Disallow: / to robots.txt for each bot you want to exclude. If you want to welcome them, you don't need to do anything extra: the standard sitemap and RSS feed are enough.
6. Monitor over time. Repeat your log analysis every quarter. The AI crawler landscape changes fast, as Cloudflare data shows.

What are the most common mistakes to avoid?

Most mistakes come from doing nothing or relying on solutions that aren't operational yet. Here are the most common ones.

One underrated mistake concerns quality: Mueller reminded us that Google doesn't use a sitemap unless it's convinced the site has new, important content worth indexing. The same principle applies to AI crawlers. A perfect sitemap on a site with outdated content won't deliver results. The topic ties directly to what we covered in our article on how Google handles fact-checking on AI content: content quality and reliability are always the starting point.

How can you tell if your setup is working?

The main signals come from server logs and, over time, from citations in AI systems. Here are the indicators to monitor.

On the topic of AI citations, it's also worth reading what's changing in the relationship between Google and sites that provide AI answers: the economics around AI visibility are evolving fast and could shape your strategic choices.

For a complete overview of every aspect of SEO and AI visibility, our topic cluster collects all the relevant updates for Italian companies.

So, is it worth optimizing sitemaps and RSS for AI crawlers?

Worth it if you have regularly updated content, want to be cited by AI systems like Perplexity or Gemini, and haven't yet checked the name and setup of your sitemap. The fix is quick and the visibility potential is real.

Somewhere in between: you have a brochure site with few updates. It's still worth setting up a standard sitemap.xml and robots.txt, but the impact will be limited until you publish more often.

Not yet if your business model depends on exclusive content you don't want to hand over to AI model training. In that case, configure robots.txt to block specific AI crawlers and check that the block works.

How we see it at SHM

To us, Mueller's news isn't a technical curiosity: it confirms that visibility in AI systems is built on the same foundations as classic SEO: structure, content quality, accessible data. There's no magic formula and no special file to add: it takes methodical auditing and configuration work that many Italian companies haven't done yet.

We believe AI should be integrated to free companies from routine, not to add complexity. Optimizing your sitemap and RSS is one of those steps you take once and that keeps paying off over time, with no ongoing upkeep.

In the SEO for AI projects we run, auditing discovery files (sitemap, robots.txt, RSS feed) is always the first step: that's where the quickest wins are. We pair it with the strategic integration of AI into content processes, to make sure what crawlers find is also worth citing. If you want to understand how your site looks to crawlers, the starting point is always a data analysis, server logs included.

Frequently Asked Questions

A sitemap isn't a technical file: it's the business card you hand to AI systems every time they publish an answer in your industry.