Contact
What are you interested in?
Log File Analysis.
Log file analysis evaluates a web server's access logs to see which pages search engine and AI crawlers actually visit, how often and with what result. It shows real crawling behaviour, not just a simulation.
Log File Analysis — Explained in Detail
Every web server records requests: time, requested address, status code, user agent and IP address. These log files contain what SEO tools can only estimate: when Googlebot fetched which page, whether it received errors in the process, and which pages it has not visited for weeks. Now that AI crawlers are active, the logs also show whether and how often GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot read a website.
Log file analysis answers questions that Search Console only partly covers: does the crawler spend its time on important pages or on parameter variants? Are new pages visited promptly? Are there sections that are not crawled at all? Does the server respond more slowly to bots than to people? For large shops and portals with a limited crawl budget in particular, this is the most reliable data source.
An example: an online shop with 8,000 products wonders why new items only appear in Google after weeks. Log file analysis shows that a large share of Googlebot's requests go to filter combinations with dynamic URLs. Once these are contained with canonicals and robots.txt, crawling shifts to product and category pages.
When log file analysis pays off
For a website with twenty pages, log file analysis is usually unnecessary; Search Console is enough. It becomes worthwhile for large websites, for indexing problems with no obvious cause, after relaunches with many redirects, and when you want to know which AI crawlers actually visit the website.
The prerequisite is access to the raw data. With some hosting packages and website builders, logs are unavailable or only available in truncated form. Log files contain IP addresses and must therefore be handled carefully under data protection law; for SEO analysis, filtered bot requests are sufficient.
Log files and AI visibility
For work on visibility in AI answers, log files are currently one of the few hard data sources. They show whether an AI provider's search crawler knows a page at all, and how often fetch agents such as ChatGPT-User retrieve pages on behalf of users. The background is explained in the entry on AI crawlers.
On our own website we record bot requests separately from human visits, without cookies. For client websites we check reachability, status codes and robots.txt in the website audit; in GEO support, checking the robots.txt permissions for AI crawlers is included. A full log file analysis is not included in either and requires access to the raw data.
Related Page
Website AuditFrequently Asked Questions About Log File Analysis
Evaluating server access logs to see which pages search engine and AI crawlers actually fetch, how often, with which status code and how quickly the server responds. It shows real crawling, not a simulation.
Specialised tools such as the Screaming Frog Log File Analyser, features in enterprise SEO platforms, or your own analyses with spreadsheets and scripts. For smaller websites, a filter on the user agents of the most important bots is often enough.
Yes, provided the logs are complete. Requests from OpenAI appear with user agents such as GPTBot, OAI-SearchBot or ChatGPT-User. The logs do not show whether a page is then quoted in an answer, but they do show whether it was read at all.
More Terms Starting with "L"
LCP (Largest Contentful Paint)
LCP is one of the three Core Web Vitals and measures how long it takes for the largest visible eleme...
Local Pack
The Local Pack (also: Map Pack or Google 3-Pack) is the block in Google search that displays three l...
Lighthouse Score
The Lighthouse Score is a rating from Google that measures the quality of a website in four categori...
Ready for Your Project?
Apply this knowledge to your website — DLM Digital will help you.