Log file analysis is reading your server’s access logs to see exactly which bots and people requested which pages, when, and what your server answered. It is the only first-hand record of how Googlebot and AI crawlers such as GPTBot, OAI-SearchBot and ClaudeBot actually treat your site. Analytics tools can’t show this, because bots don’t run their tracking code.

What a log line tells you
Every request leaves one line. A typical line holds the visitor’s IP address, the date and time, the requested URL, the HTTP status code your server returned, the response size, the referrer and the user agent, which is the name the visitor gives for itself.
Read together, those lines answer questions no other tool can: Did Googlebot crawl the page we updated last week? Is ChatGPT’s search crawler getting a 200 or an error? How much crawling goes to pages we don’t even want indexed?
Why it matters more in AI search
Google Analytics 4 runs on JavaScript in the browser. Crawlers send plain HTTP requests and never run that code, so AI bot traffic is invisible in GA4. Your logs, or your CDN’s logs, are where it shows up.
AI companies also run several bots with different jobs. Mixing them up leads to bad decisions:
- OpenAI: GPTBot collects training data. OAI-SearchBot crawls for ChatGPT search. ChatGPT-User fetches a page when a user’s question needs it.
- Anthropic: ClaudeBot collects training data. Claude-SearchBot and Claude-User do the search and user-request work.
- Perplexity: PerplexityBot crawls; Perplexity-User fetches pages for a live answer.
- Google: Google-Extended controls whether your content trains Gemini, but it is only a robots.txt token. You will never see it in your logs. Google’s crawling still shows up as Googlebot.
One warning: a user agent is just a label, and anyone can fake it. Verify important bots before you trust the numbers. Google documents a reverse DNS check for Googlebot, and OpenAI publishes the IP ranges its crawlers use.
What to check in your logs
- Status codes for each bot. A 200 means the bot got in. A 403, 429 or 503 means it showed up and was turned away.
- Crawl frequency on money pages. Your service and product pages should be crawled regularly. If bots spend their time on tag pages, filters and old URLs, that’s crawl budget you’re wasting.
- Search bots versus training bots. You can block training and still allow search. Check that your rules do what you decided.
- Fake visitors. Requests that claim to be Googlebot but fail verification are scrapers. Block those, not the real bot.
What our own logs showed
In September 2026, our server logs showed that some GPTBot and ClaudeBot requests were being turned away with 429 errors at the hosting layer. Our robots.txt allowed them. The server didn’t. Nothing in Search Console or GA4 hinted at it.
After we moved stradiji.com to a new host on Nov. 1, 2026, our own crawler, set to identify as Googlebot, started getting 403 errors. The new host blocks requests that claim to be Googlebot but don’t come from Google. That’s good protection, and it’s also a reminder: test your site with your crawler’s real name, not a borrowed one.
We also saw traffic in GA4 that looked human and even appeared to come from Google, yet kept re-crawling old posts. GA4 couldn’t tell us what it was. Logs and IP checks could.
Run through our AI skills
We ran this term through the same skills we use on client sites
At Stradiji we built our own Claude skills for SEO and GEO work. Every glossary page passes through them before it goes live. Here is what each one flagged on this page.
Two bots, two rows
OAI-SearchBot (search) and GPTBot (training) are separate decisions and are reported on separate rows. Many sites block both and cut off their own ChatGPT search visibility.
The most commercial term in our glossary
“log file analysis” gets 1,000 US searches a month with a $21.76 cost per click (Semrush, November 2026). People searching it are usually about to buy a tool or a service.
Crawl budget follows signals
Logs show where Google spends its crawl. If that doesn’t match the pages you care about, your internal links and sitemaps are sending mixed signals.
Could ChatGPT write this?
A six-part originality score with a publish threshold of 70. The 429 case above is what no generic definition can copy.
Want these skills run on your own site? Start with the free GEO Score Card below.

Log file analyzer tools
For a one-off audit, a desktop tool such as Screaming Frog Log File Analyser imports raw logs and groups requests by bot, URL and status code. Semrush has a Log File Analyzer inside Site Audit. If your site runs behind Cloudflare or Amazon CloudFront, platforms like GeoGenie can read CDN logs continuously and separate AI crawls into indexing, live answers and training.
The one thing to do this week
Ask your developer or host for the last 7 days of access logs. Search them for “OAI-SearchBot” and “Googlebot” and count the status codes. If you see anything other than 200 and 304 on pages you want in AI answers, you’ve found your first fix.
Related terms
Frequently asked questions
Where do I find my server log files?
On most hosts they’re in the control panel under “Logs” or “Access logs,” or in a folder like /var/log on your server. If you use a CDN such as Cloudflare, the CDN’s logs are often more complete than your server’s.
Can Google Analytics show AI crawler traffic?
No. GA4 only records visitors that run its JavaScript. Crawlers don’t. GA4 can show people who click through from ChatGPT or Perplexity, but not the bots themselves.
Should I block GPTBot?
It depends on whether you want your content used to train OpenAI’s models. Blocking GPTBot doesn’t block OAI-SearchBot, so you can opt out of training and still appear in ChatGPT search. Decide each one separately.
How often should I run a log file analysis?
Run it after every big change: a migration, a new CDN or firewall rule, a redesign. For large sites, a monthly check is a sensible baseline.
Sources
- OpenAI, “Overview of OpenAI Crawlers” (GPTBot, OAI-SearchBot, ChatGPT-User)
- Anthropic, crawler documentation (ClaudeBot, Claude-SearchBot, Claude-User)
- Google Search Central, “Verifying Googlebot and other Google crawlers”
- Stradiji server logs, September and November 2026
