- If you read "Googlebot" from a user agent string alone, most of what you see can be a vulnerability scanner: on a 1,782-page programmatic site we audited, 10,042 of 10,811 requests claiming to be Googlebot were fake, all 404s on files like /.env and /.git/HEAD.
- Filter by IP and reverse DNS first. Real Googlebot made 769 requests in 9 days, hit 477 distinct URLs, and left 38 real 404s that pointed to three concrete defects.
- The fix is small: correct robots.txt, remove or generate missing og:image files, and keep a sitemap index so the crawler has a reason to return.
You notice it as a crawl problem, but the data is contaminated before you look at it
The symptom is usually indirect. Pages that used to get crawled within days sit untouched for weeks. A new section of the site gets no impressions because the crawler never fetched it. Someone opens the log, greps for Googlebot, sees a healthy-looking number of hits, and concludes the crawler is active. That conclusion can be wrong by an order of magnitude.
On a 1,782-page programmatic site we audited, the access log showed 10,811 requests carrying a Googlebot user agent over nine days, from 11 to 20 September 2026. That looks like a well-crawled site. It was not. Only 769 of those requests came from Google's 66.249.* range. The remaining 10,042 came from 36 IPs with no reverse DNS, mostly inside the block 94.154.46.*. Every one of those 10,042 requests was a 404 on a path no real crawler would request: /.env, /.env.bak, /.env.local, /secrets.json, /config.php, /.git/HEAD, /docker-compose.yml. That is a vulnerability scanner wearing a Googlebot user agent. It is not crawling your content. It is looking for credentials.
The practical consequence: if you measure crawl activity from the user agent string, 93% of your dataset is noise. Any decision you make from it, about which sections are being crawled, about crawl frequency, about whether a deploy helped, is built on a scanner.
How to diagnose it with the log you already have
You do not need a paid crawler log tool for this. The nginx access log, compressed or not, is enough. The pipeline we used was four commands.
- Read the logs and isolate the user agent.
zcat access.log* | grep Googlebot. This gives you every request claiming to be Googlebot. Treat this as a candidate set, not as data. - Group by IP. Pipe into
awk '{print $1}' | sort | uniq -c | sort -rn. In a standard nginx combined log, $1 is the client IP. On the site above, this single step separated 66.249.* from 36 unrelated IPs. - Group by path and status.
awk '{print $7, $9}' | sort | uniq -c | sort -rn. $7 is the request path, $9 the status code. Fake Googlebot produces a dense cluster of 404s on dotfiles. Real Googlebot produces 200s on HTML, images and feed files, with a thin tail of 404s worth investigating. - Confirm the IP is Google. Google publishes its crawler IP ranges and supports reverse DNS: a genuine Googlebot IP resolves to a hostname in the googlebot.com or google.com domain, and that hostname resolves back to the same IP. A request from 94.154.46.* resolves to nothing.
One extra check costs nothing and catches a common mistake: if your site sits behind a CDN or reverse proxy, the client IP in the log may be the proxy, not the visitor. In that case $1 is useless and you need the real client IP from the forwarded header, configured at the server level. Confirm which one your log format records before you group anything.
What the data showed after filtering
Once the 10,042 scanner requests were removed, 769 real Googlebot requests remained. Of those, 718 returned 200, 38 returned 404, 9 returned 301 and 4 returned 405. The crawler reached 477 distinct URLs out of 1,782, roughly a quarter of the site in nine days.
By page type, the distribution was uneven. The 680 blog articles received 437 hits. The 992 programmatic city pages received 111. Assets, mostly images, received 89. Robots, sitemap, RSS and llms.txt together received 72. The home page received 24, the SEO cluster 15, and the 4 service hubs 11. The crawler was spending most of its budget on the blog and comparatively little on the programmatic section, which was the part of the site built to rank.
Daily volume sat between 20 and 40 hits per day, then jumped to 125 on 18 September and 265 on 19 September. That spike followed a deploy that added a sitemap index and rebuilt the blog hub. The crawler reacted within a day. This is the clearest signal in the dataset: crawl volume is responsive to structural changes, and a sitemap index is one of the levers that moves it.
The 38 real 404s were the actionable part, and they split into three groups:
- 16 hits on /robots.txt, on 13 and 14 September. The file answered 404 on those days. It has answered 200 since the 18 September deploy at the latest. A missing robots.txt is not fatal, but it removes your ability to point the crawler anywhere and it makes every crawl session slightly less directed.
- 3 hits on /sitemap.xlm, a typo for sitemap.xml. The misspelling was not found in the site's own HTML, so it came from an external link or an old reference. Small in volume, but it is a dead end the crawler followed three times.
- 10 hits on /og/sistema-sob-medida-<city>.jpg. The 992 city pages each declare an og:image, and those files do not exist on disk. Only 9 city OG files were ever generated. Googlebot requested them and received 404.
| Metric | Value | What it means |
|---|---|---|
| Requests with Googlebot user agent | 10,811 | Raw number, unusable without IP filtering |
| Requests from 66.249.* (real Googlebot) | 769 | 7% of the raw total |
| Requests from 36 non-Google IPs | 10,042 | Scanner traffic, all 404s on dotfiles and config paths |
| Real Googlebot responses by status | 718 x 200, 38 x 404, 9 x 301, 4 x 405 | The 38 404s are the only part worth acting on |
| Distinct URLs crawled in 9 days | 477 of 1,782 | About a quarter of the site |
| Hits on blog articles (680 pages) | 437 | Disproportionate share of the budget |
| Hits on programmatic city pages (992 pages) | 111 | Under-crawled relative to page count |
| Peak daily crawl volume | 265 on 19/09 | Followed a deploy adding a sitemap index |
The fix
Three changes, all small, all verifiable from the same log pipeline.
1. Serve a real robots.txt and point at a sitemap index. The file was 404 for at least two days. Once it exists, it should reference the sitemap index rather than a single sitemap, because the site has distinct sections with distinct update rhythms.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap-index.xml
2. Stop declaring og:image files that do not exist. There are two valid options and one invalid one. The invalid option is to leave the tag in place pointing at a 404. The valid options are to generate the missing files, or to remove the tag from pages where no image exists. Generating 983 images is a build step, not a manual task. The template that renders each city page should emit the tag only if the file is present:
<?php if (file_exists($ogImagePath)): ?>
<meta property="og:image" content="<?= $ogImageUrl ?>">
<?php endif; ?>
The same check belongs in whatever generates the sitemap, so the sitemap never lists a URL that returns 404 or a resource that has not been built.
3. Fix the /sitemap.xlm reference. It is not in the site's own HTML, so search your templates, your redirect rules, your email signatures and any external profile that links to the site. A single wrong character in one place produces a repeated dead end.
4. Keep the sitemap index fresh. The 18 September deploy produced the largest crawl day in the dataset. Whatever process rebuilt the blog hub should run on a schedule, not once. A sitemap index with accurate lastmod values gives the crawler a reason to come back and a way to prioritise.
5. Block the scanner noise at the edge, but never by user agent alone. Blocking on "Googlebot" in the user agent will block real Googlebot. Block on the IP ranges you identified, or better, verify Googlebot properly and reject everything else that claims to be it. A request from an IP with no reverse DNS claiming to be Googlebot is a request you can safely drop.
How to verify it worked
Measure the same four things before and after, from the same log pipeline, over comparable windows.
- Real Googlebot requests per day. Baseline was 20 to 40 per day, with 125 and 265 on the two days after the deploy. After the fixes, watch whether the baseline itself rises, not just the spike days.
- 404 count from real Googlebot. Baseline was 38 over nine days, split 16 / 3 / 10 across the three defects. The target is not zero, some 404s are normal, but the three known groups should disappear.
- Distinct URLs crawled. Baseline was 477 of 1,782 in nine days. Watch whether the programmatic section, which received 111 hits against 992 pages, starts receiving a larger share.
- Status distribution. Baseline was 718 x 200 against 38 x 404. The ratio should improve as the missing og:image files stop being requested or stop being declared.
Run the same commands, same window length, same filters. If you change the pipeline between measurements, you are comparing two different things. Keep the IP filter in place permanently: the scanner will keep sending requests, and it will keep inflating any unfiltered number you look at.
Frequently asked questions
Why can a scanner use the Googlebot user agent without being blocked?
User agent strings are self-declared. Any client can send any string. The only reliable way to identify Googlebot is to check the source IP against Google's published ranges, or to resolve the IP to a hostname in the googlebot.com domain and confirm the hostname resolves back to the same IP. A request from an IP with no reverse DNS that claims to be Googlebot is not Googlebot.
Is a missing robots.txt actually a crawl problem?
It is a smaller problem than it looks, and a real one. Without the file, the crawler has no sitemap reference and no explicit directives. In the dataset above, real Googlebot requested /robots.txt 16 times across two days and received a 404 each time. That is not catastrophic, but it is a wasted request and a missing signal, and it is fixed in minutes.
Do og:image 404s affect indexing of the page itself?
The page can still be indexed. The problem is that the crawler spends budget fetching a resource that does not exist, and any surface that renders the tag, social previews and some aggregation features, gets nothing. On the site above, 992 pages declared an image and only 9 files existed. That is a systematic waste, not an isolated bug.
How often should crawl data be checked?
After every deploy that changes site structure, sitemaps or internal linking, because the response can be visible within a day. The 18 September deploy produced 125 hits that day and 265 the next, against a baseline of 20 to 40. Outside of deploys, a monthly pass over the same four metrics is enough to catch regressions before they compound.
For the wider method behind this kind of log analysis, including how the checks fit into a technical SEO audit, see the technical SEO page at Prospectia.
Frequently asked questions
User agent strings are self-declared. Any client can send any string. The only reliable way to identify Googlebot is to check the source IP against Google's published ranges, or to resolve the IP to a hostname in the googlebot.com domain and confirm the hostname resolves back to the same IP. A request from an IP with no reverse DNS that claims to be Googlebot is not Googlebot.
It is a smaller problem than it looks, and a real one. Without the file, the crawler has no sitemap reference and no explicit directives. In the dataset above, real Googlebot requested /robots.txt 16 times across two days and received a 404 each time. That is not catastrophic, but it is a wasted request and a missing signal, and it is fixed in minutes.
The page can still be indexed. The problem is that the crawler spends budget fetching a resource that does not exist, and any surface that renders the tag, social previews and some aggregation features, gets nothing. On the site above, 992 pages declared an image and only 9 files existed. That is a systematic waste, not an isolated bug.
After every deploy that changes site structure, sitemaps or internal linking, because the response can be visible within a day. The 18 September deploy produced 125 hits that day and 265 the next, against a baseline of 20 to 40. Outside of deploys, a monthly pass over the same four metrics is enough to catch regressions before they compound.