Googlebot (Crawler).

A crawler works like a tireless visitor. It starts with known addresses, fetches the page, reads all the links on it and adds new addresses to its queue. In this way it gradually discovers new and changed pages. Google runs several crawlers for this; the most important is Googlebot Smartphone, because since the switch to the mobile-first index Google assesses websites from the perspective of a smartphone.

After fetching comes rendering at Google: the Web Rendering Service runs the page in a current version of Chromium, including JavaScript, to see the final content. This step can happen with a delay. Content that only appears after clicks, while scrolling or after logging in is usually not seen by Googlebot. What it does see is analysed and, if nothing speaks against it, indexed.

Website owners can control crawling. The robots.txt specifies which areas should not be fetched; the noindex directive in the meta robots tag or in the HTTP header prevents inclusion in the index; the XML sitemap lists the important URLs. The difference matters: a page blocked in robots.txt can still appear in the index if other pages link to it, and a noindex directive only works if the crawler is allowed to fetch the page.

Besides Google, many other crawlers visit a website: Microsoft's Bingbot, crawlers from SEO tools and, increasingly, AI crawlers such as GPTBot or PerplexityBot. Some programs falsely pose as Googlebot. Genuine Googlebot requests can be verified using a reverse DNS lookup or the IP ranges published by Google.