Contact
What are you interested in?
AI Crawlers.
AI crawlers are bots run by AI providers that fetch websites in order to train language models or to retrieve content for answers in real time. Examples are GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, PerplexityBot and ClaudeBot. Website owners use robots.txt to control which of these crawlers get access.
AI Crawlers — Explained in Detail
The rise of AI assistants has created a new group of crawlers. They can be roughly divided into three kinds. Training crawlers such as GPTBot (OpenAI) or ClaudeBot (Anthropic) collect content with which future language models are trained. Search crawlers such as OAI-SearchBot (OpenAI) or PerplexityBot build an index from which AI search selects sources for answers. User fetchers such as ChatGPT-User, Claude-User or Perplexity-User retrieve a page when a person specifically asks about it in a conversation or shares a link.
The distinction matters because blocking has different consequences. Blocking GPTBot prevents content from flowing into OpenAI's future training data but keeps you findable in ChatGPT search via OAI-SearchBot. Blocking OAI-SearchBot means you do not appear there as a source. Google does not use a separate crawler for AI features but a dedicated token in robots.txt: Google-Extended controls whether content may be used for training and grounding Gemini models; AI Overviews in Google Search, by contrast, depend on normal indexing by Googlebot.
Technically, many AI crawlers have an important limitation: they execute little or no JavaScript. Content that a website only builds in the browser via JavaScript remains invisible to them. Websites that deliver their content from the server (server-side rendering) or generate it statically have a clear advantage here. Files such as llms.txt can additionally provide a compact overview of the most important content.
Whether to allow AI crawlers is a business decision. Media companies often block training crawlers to protect their content. For most SMEs, the benefit of being named in AI answers outweighs this; they allow at least the search and user crawlers. What matters is that the decision is made deliberately and not taken by accident through a blanket blocking rule from a hosting provider or a firewall.
Controlling AI crawlers in robots.txt
Each crawler identifies itself with its own user agent and can be addressed individually in robots.txt, for example with 'User-agent: GPTBot' followed by 'Allow: /' or 'Disallow: /'. The same notation applies to Google-Extended, even though it is a token and not a separate bot. robots.txt is a request, not a technical barrier; reputable providers say they comply with it.
It is also worth looking at firewall and CDN settings. Some services block AI bots by default or show them a challenge page. In that case, a welcoming robots.txt is of no use. Server logs show whether the bots actually get through.
Example: a firewall blocks ChatGPT
An architecture firm wonders why ChatGPT only gives outdated details from directories when asked about the firm. The check shows that robots.txt allows all bots, but the hosting provider's protection service answers requests from OAI-SearchBot and ChatGPT-User with a challenge page. The AI systems therefore never see the website.
After the firewall rules are adjusted, the pages are reachable. Our free AI visibility check tests whether your own website is readable for AI crawlers; ongoing monitoring of whether a company is named in AI answers is part of GEO management from CHF 300 per month.
Related Page
robots.txtFrequently Asked Questions About AI Crawlers
That is a business decision. GPTBot collects training data; blocking it prevents your content from being used for future models but does not directly affect whether you appear in ChatGPT search. OAI-SearchBot is responsible for that. Most SMEs allow both in order to be present in AI answers.
A robots.txt token with which website owners control whether Google may use content for training and grounding Gemini models. It is not a separate crawler. Appearing in AI Overviews in Google Search depends on normal indexing.
Many do not, or only to a limited extent. They often do not see content that is only created in the browser via JavaScript. Websites with server-delivered HTML are considerably easier for AI crawlers to read.
More Terms Starting with "K"
Keyword
A keyword (search term) is the word or phrase that a user enters into a search engine. Keywords are ...
KPI (Key Performance Indicator)
KPIs are measurable metrics that quantify the success of a measure or strategy. In digital marketing...
Knowledge Graph
The Knowledge Graph is Google's knowledge base, which records real-world entities such as people, co...
Ready for Your Project?
Apply this knowledge to your website — DLM Digital will help you.