Site Not Cited by ChatGPT? Our Seven-Step Troubleshooting Log from a Near-Misdiagnosis

Limitations first

• Our self-built instrument only came online at 2026-08-10 07:52 UTC, so the sample is very young. Numbers from the instrument describe only the world after that moment.
• The edge-side reconciliation used Cloudflare GraphQL's 72-hour window. Anything outside that window was not examined this time.
• This is a log from a single website, not industry statistics. The checklist can be reused; the numbers cannot be extrapolated.
• A 204 response to a sitemap submission means "accepted", not "indexed". IndexNow's 200/202 are likewise just delivery receipts.

A zero that almost went into a report

It all started with a zero.

We built an origin-side instrument for our own site that logs every crawler visit. We opened it and saw: OAI-SearchBot — the crawler OpenAI uses to index content for ChatGPT search — 0 visits.

The first reaction was the natural one: "OpenAI has never been here. No wonder ChatGPT doesn't cite us." That sentence very nearly went into a formal report.

Fortunately we took one more step: we pulled the 72-hour ground truth from Cloudflare's edge via GraphQL and reconciled. The result overturned everything —

• OAI-SearchBot: 5 visits, all 200 (content successfully fetched)
• GPTBot: 10 visits
• Claude family: 6 visits (ClaudeBot 1 + Claude-User 2 + anthropic 3)
• Amazonbot: 151 visits
• meta-external: 423 visits

Not only had they been here — they got what they came for every time. So what about that zero? The answer is not flattering: our instrument only came online on 08-10, and OAI-SearchBot's visits happened before it was switched on.

An analogy: you finally install a door camera at midnight, review the footage the next morning, and announce that the mail carrier has never visited your house. The camera is not broken; your observation window is just too short.

In hindsight, this mistake is especially easy to make, for an ironic reason: we built the instrument ourselves, so we trusted it the most. The more a tool is your own handiwork, the easier it is to forget to ask first: "Since when has it been watching?"

The cost of the misdiagnosis would not have been cheap, either. Had "OpenAI never came" gone into the report, every prescription that followed would have been wrong: rewrite the content, tweak the titles, blame the algorithm — while the real problem was that we had misread our own dashboard.

That near miss forced us to turn the whole process into a checklist. The seven steps below are the version we walked through ourselves, with hands-on evidence at every step. The order matters: from outside in, from cheap to expensive. If an earlier step fails, no conclusion from a later step holds.

The seven-step checklist

Step 1: Is robots.txt keeping visitors out? Do the cheapest check first. Open robots.txt and go line by line looking for Disallow rules that hit AI crawlers. Read it word by word, do not skim: who is blocked, on which paths, and whether the whole site is blocked. We passed this step. But many sites are eliminated right here — one line left behind by whoever built the site years ago amounts to hanging a "no soliciting" sign on your own door, then complaining that nobody visits.

Step 2: Is your CDN or edge blocking on your behalf? Check the actual settings — do not guess. Even if robots.txt opens the door, the CDN may be closing it somewhere you do not know about. In recent years CDNs have been rolling out "AI crawler protection" features; some defaults change across versions, and some are switched on by a well-meaning hosting agency — without your knowledge. We checked our Cloudflare settings directly: ai_bots_protection=disabled, crawler_protection=disabled, no WAF rules intercepting — the door was fully open. The point is not our result but the method: this step must be done by opening the settings page or calling the API. "I don't remember turning it on" is not evidence.

Step 3: Reconcile the edge against your own instrument. The time-window trap lives here. This is the step where we fell. Same website: the origin instrument said 0, the edge said 5 visits, all 200. Neither is lying — they measure different times at different positions. In practice: before comparing anything, write down in plain words "when my instrument came online" and "how long the edge query window is", and only then pull the numbers. Without those windows written down first, any comparison is wrong; once you write them down, you will usually see your own hole before you even reconcile.

Step 4: Verify at the index layer. Being crawled does not mean being known. A crawler's visit only means content was hauled away; whether a search engine's index knows you is a separate ledger. It is like a bookstore: someone leafing through a book in the shop, and that book being listed in the library's catalog, are unrelated facts. We took 3 pages to Google Search Console for URL inspection; all 3 came back "URL is unknown to Google". In other words, at Google's index layer we effectively did not exist — a question that must be checked separately from whether crawlers visited.

Step 5: Was the sitemap "actually" submitted? Digging further, an even less flattering truth: the entire domain had 0 sitemaps in GSC's list — never submitted. The corresponding result: 0 impressions over 28 days. We had assumed that going live meant being seen; the fact is we had never checked in at the front desk. The key word is "actually": do not ask your memory, open the submission list in the console. The fix was completed the same day: 6 sitemaps submitted, all accepted with 204; IndexNow pinged as well, returning 200/202. A reminder again: accepted means received, not indexed.

Step 6: External mentions. Is anyone talking about you? AI citation does not depend only on whether the road to your own door is open; it also depends on whether people out there mention you. Do other articles link to you, do industry directories list you, do communities talk about you — these are the circumstantial evidence AI uses to judge whether a source is worth citing. For this step we have no new numbers within this window, only a qualitative reminder from our previous ledger: five months, roughly 30 million fetches, against just 476 human clicks (a visible lower bound) (https://km.idaeo.ai/ai/ai-crawler-shop-ledger). Being hauled away a lot does not mean being talked about or cited.

Step 7: Scheduled re-probing. One check is only a snapshot. Troubleshooting does not end once you finish it once. Index-type crawlers run on a weekly rhythm: our previous article measured a median first crawl of 4.2 hours for the training-type ClaudeBot, versus 332.8 hours (13.9 days) for the index-type OAI-SearchBot (https://km.idaeo.ai/ai/ai-crawler-marketing). Fix your door today, and the slowest visitors will only pass by again on a timescale of weeks. Schedule re-checks to match that rhythm: set a fixed review date, rerun the same checks when it arrives, and keep the results on file so your future self has something to compare against — instead of flipping the table when nothing moves the next day.

The core lesson

The window of your measuring tool decides the world you see.

A 0 on the instrument only proves "nothing was seen inside its window after it came online" — it does not prove "nothing happened in the world". What the edge sees, what the instrument sees, and what the index knows are separate ledgers; reconcile them separately.

We almost used a single 0 to convince ourselves that OpenAI had never come. What we should have said is: "Nothing was seen within our window; for anything outside it, ask the edge." That sentence is longer, but it is the true one.

So next time, before complaining that AI never cites you, walk through this checklist first: did your own door close on you, how long has your measurement window been open, did you check in at the front desk. Most of the time the answer is not inside the algorithm's black box — it is on your own dashboard.