robots.txt.

The file sits at /robots.txt in the root directory and is the first thing a well-behaved crawler requests when it arrives. Directives such as User-agent, Disallow and Allow define which paths a given bot is permitted to visit. Most files close with a Sitemap line pointing to the XML sitemap, which is the simplest way to make sure a crawler finds every URL you care about. The syntax is deliberately plain: one group per user agent, one rule per line.

One pitfall trips up almost everyone. robots.txt blocks crawling, not necessarily indexing. A page that is disallowed but linked from elsewhere can still end up in the index — listed without a snippet, because Google was never allowed to read it. If you need a page kept out of search results reliably, use a meta robots 'noindex' tag or password protection instead, and leave the page crawlable so that the noindex can actually be read.

A typical use: an online shop blocks its basket, internal search results and admin areas from crawling so that Google spends its crawl budget on the product and category pages that matter. In practice that is two lines — 'User-agent: *' followed by 'Disallow: /basket/' — and it blocks every bot for that path. The same logic applies to filter URLs and endless faceted navigation, which can otherwise generate thousands of near-identical addresses.

Use the file sparingly and deliberately. A stray 'Disallow: /' can shut an entire website out of the index, and that mistake usually surfaces weeks later as a collapse in traffic. After every change, test the result — the robots.txt report in Google Search Console shows exactly how Google parses your file, and the URL inspection tool confirms whether a specific address is blocked. Five careful lines beat fifty defensive ones.