What Is Log File Analysis? How to See Google and AI Crawlers on Your Site

Log file analysis is reading your server’s access logs to see exactly which bots and people requested which pages, when, and what your server answered. It is the only first-hand record of how Googlebot and AI crawlers such as GPTBot, OAI-SearchBot and ClaudeBot actually treat your site. Analytics tools can’t show this, because bots don’t run their tracking code.

Log file analysis: bots arrive at a gate and every visit is written to a record

Last updated: Nov. 4, 2026 · By Mert Erkal, founder of Stradiji

What a log line tells you

Every request leaves one line. A typical line holds the visitor’s IP address, the date and time, the requested URL, the HTTP status code your server returned, the response size, the referrer and the user agent, which is the name the visitor gives for itself.

Read together, those lines answer questions no other tool can: Did Googlebot crawl the page we updated last week? Is ChatGPT’s search crawler getting a 200 or an error? How much crawling goes to pages we don’t even want indexed?

Why it matters more in AI search

Google Analytics 4 runs on JavaScript in the browser. Crawlers send plain HTTP requests and never run that code, so AI bot traffic is invisible in GA4. Your logs, or your CDN’s logs, are where it shows up.

AI companies also run several bots with different jobs. Mixing them up leads to bad decisions:

  • OpenAI: GPTBot collects training data. OAI-SearchBot crawls for ChatGPT search. ChatGPT-User fetches a page when a user’s question needs it.
  • Anthropic: ClaudeBot collects training data. Claude-SearchBot and Claude-User do the search and user-request work.
  • Perplexity: PerplexityBot crawls; Perplexity-User fetches pages for a live answer.
  • Google: Google-Extended controls whether your content trains Gemini, but it is only a robots.txt token. You will never see it in your logs. Google’s crawling still shows up as Googlebot.

One warning: a user agent is just a label, and anyone can fake it. Verify important bots before you trust the numbers. Google documents a reverse DNS check for Googlebot, and OpenAI publishes the IP ranges its crawlers use.

What to check in your logs

  1. Status codes for each bot. A 200 means the bot got in. A 403, 429 or 503 means it showed up and was turned away.
  2. Crawl frequency on money pages. Your service and product pages should be crawled regularly. If bots spend their time on tag pages, filters and old URLs, that’s crawl budget you’re wasting.
  3. Search bots versus training bots. You can block training and still allow search. Check that your rules do what you decided.
  4. Fake visitors. Requests that claim to be Googlebot but fail verification are scrapers. Block those, not the real bot.

What our own logs showed

In September 2026, our server logs showed that some GPTBot and ClaudeBot requests were being turned away with 429 errors at the hosting layer. Our robots.txt allowed them. The server didn’t. Nothing in Search Console or GA4 hinted at it.

After we moved stradiji.com to a new host on Nov. 1, 2026, our own crawler, set to identify as Googlebot, started getting 403 errors. The new host blocks requests that claim to be Googlebot but don’t come from Google. That’s good protection, and it’s also a reminder: test your site with your crawler’s real name, not a borrowed one.

We also saw traffic in GA4 that looked human and even appeared to come from Google, yet kept re-crawling old posts. GA4 couldn’t tell us what it was. Logs and IP checks could.

Run through our AI skills

We ran this term through the same skills we use on client sites

At Stradiji we built our own Claude skills for SEO and GEO work. Every glossary page passes through them before it goes live. Here is what each one flagged on this page.

geo-masterApplied

Two bots, two rows

OAI-SearchBot (search) and GPTBot (training) are separate decisions and are reported on separate rows. Many sites block both and cut off their own ChatGPT search visibility.

semrush-data-analystOpportunity

The most commercial term in our glossary

“log file analysis” gets 1,000 US searches a month with a $21.76 cost per click (Semrush, November 2026). People searching it are usually about to buy a tool or a service.

google-seo-goldmineApplied

Crawl budget follows signals

Logs show where Google spends its crawl. If that doesn’t match the pages you care about, your internal links and sitemaps are sending mixed signals.

non-commodity-content77/100

Could ChatGPT write this?

A six-part originality score with a publish threshold of 70. The 429 case above is what no generic definition can copy.

Want these skills run on your own site? Start with the free GEO Score Card below.

Which AI bot does what: OpenAI, Anthropic, Perplexity and Google crawlers, plus a log file audit checklist

Bots arrive at your server’s door. Most get in, some are turned away, and the log records both.

Log file analyzer tools

For a one-off audit, a desktop tool such as Screaming Frog Log File Analyser imports raw logs and groups requests by bot, URL and status code. Semrush has a Log File Analyzer inside Site Audit. If your site runs behind Cloudflare or Amazon CloudFront, platforms like GeoGenie can read CDN logs continuously and separate AI crawls into indexing, live answers and training.

The one thing to do this week

Ask your developer or host for the last 7 days of access logs. Search them for “OAI-SearchBot” and “Googlebot” and count the status codes. If you see anything other than 200 and 304 on pages you want in AI answers, you’ve found your first fix.

Related terms

Frequently asked questions

Where do I find my server log files?

On most hosts they’re in the control panel under “Logs” or “Access logs,” or in a folder like /var/log on your server. If you use a CDN such as Cloudflare, the CDN’s logs are often more complete than your server’s.

Can Google Analytics show AI crawler traffic?

No. GA4 only records visitors that run its JavaScript. Crawlers don’t. GA4 can show people who click through from ChatGPT or Perplexity, but not the bots themselves.

Should I block GPTBot?

It depends on whether you want your content used to train OpenAI’s models. Blocking GPTBot doesn’t block OAI-SearchBot, so you can opt out of training and still appear in ChatGPT search. Decide each one separately.

How often should I run a log file analysis?

Run it after every big change: a migration, a new CDN or firewall rule, a redesign. For large sites, a monthly check is a sensible baseline.

Sources

GEO · Stradiji Analysis

Is your brand visible in ChatGPT, Gemini and Perplexity?

Take the free interactive score card built by Stradiji, an SEO consultancy serving global clients since 2010. 38 checks across 3 doors. See your weakest door and the gaps to close first.

253 max points
Calculate Score →
WhatsApp'tan yazın