Crawl Budget.

The crawl budget describes how many URLs Googlebot retrieves from a website in a given period. It follows from two factors: crawl capacity, meaning how much traffic the server tolerates before it slows down, and crawl demand, meaning how important and how fresh Google considers the content. Neither figure is published as a number you can look up; what you can observe are the crawl statistics in Search Console, which show how the two play out for your own site over time.

For small and medium sites with a few hundred pages the crawl budget is rarely a bottleneck — Google generally crawls them completely. It becomes relevant on large properties with many thousands of URLs, such as sizeable online shops, news portals or programmatically generated pages. There, wasted budget can mean that important pages are revisited less often, so price changes, new stock or updated articles take longer to show up in search results than they should.

The usual budget consumers are endless URL parameter combinations, filter navigation with millions of variants, duplicate content, soft 404 pages and slow server responses. Defusing those traps with robots.txt, clean canonicals, faster response times and a lean site structure steers the budget towards the pages that are actually meant to rank. An accurate XML sitemap helps as well, because it tells Google which URLs you consider worth revisiting. None of these measures improve rankings by themselves — they simply stop attention being spent on addresses that were never meant to be found.

An example: a shop with 50'000 products and countless additional filter URLs can flood Google with parameter pages of no value. Consolidating those through robots.txt and canonicals leaves more budget for the genuine product and category pages, which are then re-indexed faster and more often. Nothing about the content changed — only how much of Google's attention reached it. In Search Console the effect usually shows up as a shift in the crawl statistics well before it shows up in rankings, which makes it one of the few technical measures with an early feedback signal.