Contact
What are you interested in?
ai.txt.
ai.txt is a text file in the style of robots.txt that specifically signals to AI crawlers and agents which content they may read, summarise and cite.
ai.txt — Explained in Detail
ai.txt is a file placed in the web root (/ai.txt) that gives AI crawlers rules in the familiar robots.txt style: User-Agent entries for bots such as GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and CCBot, each with Allow or Disallow.
Unlike robots.txt, which primarily addresses classic search engines, ai.txt is explicitly aimed at AI systems. The file often also points to llms.txt, the sitemap and a webmcp.json manifest — as a bundled signal: 'This content may be used and cited by AI.'
Like llms.txt, ai.txt is a young convention that is not yet formally standardised. But it is a clear, low-cost signal to the agent generation and part of an agent-ready website.
Related Page
llms.txtFrequently Asked Questions About ai.txt
robots.txt controls classic search-engine crawlers, while ai.txt is aimed specifically at AI crawlers and agents (GPTBot, ClaudeBot, PerplexityBot, etc.). In practice the two complement each other; many websites serve both files.
ai.txt can restrict access for certain AI bots via Disallow, but it is a voluntary convention — compliance depends on the respective provider. If you want to be visible in AI answers, allow the relevant bots explicitly via Allow.
More Terms Starting with "A"
Agent-Ready Website
An agent-ready website is built so that AI agents can not only read its content but also perform act...
AEO (Answer Engine Optimization)
AEO is the optimization of website content so that it appears as direct answers in AI-powered search...
AI Agents
AI Agents are autonomous AI systems that can independently plan, execute, and optimize tasks — witho...
Ready for Your Project?
Apply this knowledge to your website — DLM Digital will help you.