AI search crawlers allowed
Weight 3ChatGPT search (OAI-SearchBot, ChatGPT-User), Perplexity (PerplexityBot), Claude (Claude-SearchBot, Claude-User), Microsoft Copilot (Bingbot) and Google’s AI Overviews (Googlebot) fetch their sources with their own crawlers. If robots.txt blocks one of them, that service cannot cite your site.
Rating: We read your robots.txt according to the RFC 9309 standard and check, for each of these crawlers, whether the homepage is blocked. Fine: none blocked. Problem: at least one blocked. Blocks by a firewall (e.g. Cloudflare bot protection) are not visible to this check.
Indexing & snippets allowed
Weight 3With “noindex” the page drops out of the index that AI search also draws on. “nosnippet” or “max-snippet:0” explicitly forbid Google from showing your text in results and AI Overviews.
Rating: We read the “robots” / “googlebot” meta tag and the “X-Robots-Tag” HTTP header of the homepage. Problem: noindex, nosnippet or max-snippet:0 set. Otherwise fine.
Content readable without JavaScript
Weight 2Many AI crawlers only load the HTML and do not run JavaScript. If your text is only built in the browser, they see an almost empty page.
Rating: We count the words in the delivered HTML (without scripts, styles, SVG). Fine: 250 words or more. Could be better: 80 to 249. Problem: under 80.
Sitemap with change dates
Weight 1An XML sitemap shows search engines all important pages. The “lastmod” field says when a page changed — Microsoft Bing, whose index Copilot uses, explicitly recommends it so new content is picked up faster.
Rating: We look for the sitemap via the robots.txt entry, otherwise at /sitemap.xml and /sitemap_index.xml. Fine: sitemap with “lastmod”. Could be better: sitemap without “lastmod”. Problem: no sitemap found.
llms.txt
Weight 1llms.txt is a proposed standard: a short Markdown file in the root directory that tells language models who you are and which pages matter. To be honest: none of the major providers has confirmed that they use it for answers. The effort is small, though, so the item only counts once.
Rating: We fetch /llms.txt. Fine: a text file with a title line (“# Name”) and at least one link, as described in the proposal. Could be better: file missing or not in that format.
AI training crawlers (info only)
InfoGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent collect text for training language models. Blocking them is a legitimate choice and does not stop AI search from citing you. When blocked, though, the model tends to know your company only from third-party sources.
Rating: Information only, not part of the score. We show which training crawlers your robots.txt blocks.