noindex.

Not every page of a website belongs in the search results. A thank-you page after submitting a form, the results of the internal search, a customer login or a test page bring searchers no benefit and can even confuse them. With the noindex directive, a website tells search engines that a page may be fetched but should not appear in the index.

There are two ways to set noindex. In the HTML head of a page there is a meta tag with the name robots and the content noindex. Alternatively, the server sends an HTTP header X-Robots-Tag: noindex, which also works for files such as PDFs that have no HTML head. Both variants can be combined with further directives, such as nofollow (do not follow links) or noarchive.

The most common misconception concerns the interplay with robots.txt. Blocking a page in robots.txt prevents the crawler from fetching it. But then it also does not see the noindex directive. If the page is linked from elsewhere, it can still end up in the index, usually without a description. If you want to remove a page reliably from the index, you therefore have to let it be crawled and set noindex.

The most dangerous mistake is a forgotten noindex. Many systems set a global noindex on test or staging environments, and it is not removed at launch. The new website is then invisible to Google, often unnoticed for weeks. Likewise, a single tick box in a CMS plugin can remove an entire category from the index. Search Console reports such pages in the indexing report as 'Excluded by noindex tag'.