What Do AI Crawlers Check First? In a Day and a Half: robots.txt 16 Times, sitemap 3 Times, llms.txt 0

Lately the same question keeps coming up: should you write an llms.txt for AI? Meanwhile, every "AI-era website checklist" keeps getting longer: robots.txt, sitemap, llms.txt, RSS — nobody dares skip an item. The longer the checklist grows, the more you need to know: which of these files do machines actually read? Rather than trading guesses, watch where the machines' hands actually reach. We treated our own website as an instrument and logged, entry by entry, every fetch of these machine files. The answer is blunt: crawlers rush to read the notice at the door, occasionally flip through the floor map, and nobody picks up the hardcover guidebook printed specially for AI.

How we measured, and how young the sample is

By instrument we mean: no surveys, no hearsay — just our own server's access log, entry by entry. Who came, which file they took, how many times. Machines are not polite, and they do not lie. Our instrument went live at 2026-08-10 07:52 UTC. As of this article's tally, the window is only a day and a half, and the subject is a single website. That means every number below answers only for "this site, inside this window." The sample is young, and we choose to say that clearly before drawing any conclusion. This is the standing habit of this column: show the instrument and its limits first, then the numbers.

A day-and-a-half roll call of machine files

Fetch counts for the four machine files:

• robots.txt: 16 fetches
• sitemap.xml: 3 fetches
• llms.txt: 0 fetches
• RSS: 0 fetches

The roll call, bot by bot: YandexBot 12, Facebook 2, Googlebot 2, GPTBot 1, OAI-SearchBot 1, Bingbot 1.

The detail most worth noting: OpenAI's two crawlers came once each, and both fetched robots.txt — not llms.txt.

The roster is more interesting than the ranking

The most diligent reader of machine files is not the AI star you might expect. YandexBot leads by a wide margin with 12 fetches; Facebook and Googlebot follow with 2 each. The three most directly tied to AI retrieval — GPTBot, OAI-SearchBot, Bingbot — came once each.

This roster is a reminder of two things. First, "who reads your machine files diligently" and "who brings you readers" are two different questions; fetch frequency is not influence. Second, AI crawlers are not staying away — they came, they just took robots.txt. In other words, don't comfort yourself with "the AI crawlers are visiting," and don't scare yourself with "they crawl a lot." Watch what they reach for, and what you get once they've taken it.

In shopkeeper's terms

To continue the metaphor from our earlier ledger piece: your website is a shop. robots.txt is the notice at the door, stating where visitors may and may not go. sitemap.xml is the floor map. llms.txt is a hardcover guidebook made specially for AI VIPs. RSS is the sign-up sheet for new-arrival notices.

The reality of this day and a half: the door notice was read 16 times, the floor map was opened 3 times, the hardcover guidebook went untouched, and the sign-up sheet stayed blank. What you spend the most effort preparing is not necessarily what visitors reach for.

llms.txt at zero is no real surprise

Our position first: we are not against publishing an llms.txt — its cost is minimal. What we are against is placing it at the front of the checklist, treating it as the ticket that makes you visible to AI.

Google's official documentation states plainly that its AI features do not require websites to provide additional AI text files (developers.google.com/search/docs/appearance/ai-features). In other words, the search giant itself declares it does not read such files. And in our shop, OpenAI's crawlers came and looked only at the door notice. The official statement and the machines' behavior agree — this time.

The case for llms.txt is mostly "it might be useful later." It might. But "might be useful later" should not be scheduled ahead of "certainly matters now" — that is a question of priority, not of belief.

A sitemap's value is not in its read count

The floor map was opened only 3 times, but don't let that fool you. We ran a before-and-after comparison on this site: before the sitemap was submitted, Google's answer for our pages was "URL is unknown"; only after submission were they accepted. A sitemap's use is not how often shoppers browse it, but whether the building's management is willing to list your stall in the directory. A low read count does not mean low value — keep those two accounts separate.

And don't forget the big picture from our earlier ledger: over five months, AI companies fetched this site roughly 30 million times, in exchange for 476 human clicks (a lower bound of what we can see; details at https://km.idaeo.ai/ai/ai-crawler-shop-ledger ). What crawlers are truly busy with is hauling your content — not reading the extra files you wrote for them.

Limits: this window is still small

• The window is only a day and a half; the sample is young by nature.
• We observed one site only; it cannot speak for anyone else's shop.
• The 0 for llms.txt is an observation "within this site's window," not a universal claim that "nobody reads llms.txt." Other sites and other windows may differ.
• This article counts machine-file fetches only; the crawling of regular content pages is a separate ledger, outside this article's scope.

The window keeps accumulating. When the numbers change, we will update.

Action: do the basics, in the right order

This order is not a matter of taste — it follows how often each file was actually fetched on this site.

Step one: take care of robots.txt. It is the most-fetched file; make sure it is reachable and its rules are correct, so the door notice doesn't turn away the wrong visitors by itself.

Step two: build your sitemap and submit it. The point is "accepted after submission," not the read count.

Step three: add llms.txt if you wish — it costs very little. Just don't place it ahead of the first two steps, and don't expect traffic from it.

Spend the effort you save on writing content worth citing by AI. Machine files are the storefront; content is the goods.