Are AI Bots Actually Crawling Your Site? Your Server Logs Know

ai-crawler-log-analysis

There’s a gap in how almost everyone approaches AI search visibility, and it’s a strange one: we spend enormous effort trying to get cited by ChatGPT, Perplexity and Gemini without ever checking whether their crawlers have visited the site.

robots.txt tells you what you asked bots to do, but it says nothing about what they actually did. A site can allow every AI crawler and receive almost no AI crawling. A site can block GPTBot and find Bytespider crawling straight through anyway. The only way to know is to check your server logs. Because logs are awkward to work with, hardly anyone checks them.

Our free AI Crawler Log Analyzer exists for that: upload a server log and see GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest, with hits over time. As far as we can tell nobody else offers this free, which is odd, because it’s the only way to replace guesswork with evidence.

TL;DR

Server logs are the only reliable record of which AI crawlers actually visited your site, how often, and which pages they fetched. robots.txt only records your instructions, and analytics doesn’t capture bots at all. Upload a log to the free AI Crawler Log Analyzer to see real activity by user agent over time. Two findings consistently matter: AI crawlers fetch JavaScript but generally don’t execute it, and a meaningful share of requests claiming to be AI bots are spoofed.

→ Upload a log and see your AI crawler activity

Why can’t I just check robots.txt or analytics?

Because robots.txt is a request, not a record, and analytics tools deliberately filter out bots — neither one can tell you whether GPTBot visited your site yesterday. Logs are the only source that captures every request, including the ones no JavaScript tag ever fires on.

Here is what each of the three sources can actually tell you:

SourceTells youCan’t tell you
robots.txtWhat you asked bots to doWhether they did it
Analytics (GA4)Human sessions, some AI referralsBot crawling — bots don’t run JS tags
Server / CDN logsEvery request, by user agent, with timestampsWhether a crawl led to a citation

Our free robots.txt AI Crawler Checker handles the first column. It parses your rules the way engines actually resolve them, including the named-group precedence most checkers get wrong. The log analyzer handles the third column. They work as a natural pair: one shows intent, while the other shows reality.

Which AI crawlers should I look for?

The crawlers that matter in 2026 split into training crawlers, search indexers and real-time fetchers. The distinction determines whether a visit can ever send you traffic. Blocking the wrong category costs you citations, while allowing the wrong one feeds training models with no return.

User agentOperatorPurpose
GPTBotOpenAITraining
OAI-SearchBotOpenAISearch indexing
ChatGPT-UserOpenAIReal-time fetch for a user
ClaudeBotAnthropicTraining
Claude-SearchBotAnthropicSearch indexing (newer)
Claude-UserAnthropicReal-time fetch
PerplexityBotPerplexitySearch indexing
Perplexity-UserPerplexityReal-time fetch
Google-ExtendedGoogleGemini training/grounding
meta-externalagentMetaTraining
BytespiderByteDanceTraining
CCBotCommon CrawlOpen training dataset

A real-time fetch from ChatGPT-User or Perplexity-User is the most interesting line in your log: it means a human asked a question and the assistant went to your page to answer it. That’s a citation in progress.

What is the crawl-to-refer ratio, and should I care?

The crawl-to-refer ratio compares how many pages an AI operator crawls against how many visits it sends back. Across 2026, published figures have been extraordinarily volatile, which is precisely why your own data matters more than any industry average.

Reported ratios for Anthropic’s crawlers across 2026 have ranged from roughly 988:1 to well over 30,000:1 depending on the source, the month and the measurement window. OpenAI’s have similarly swung between the low hundreds and well over a thousand. Google’s traditional search sits around 5:1 by comparison.

(Those ranges come from Cloudflare Radar-derived reporting and several independent log studies published through 2026. It is worth re-checking these against current sources before quoting specific figures because they move fast.)

We’re not going to lead with a dramatic single number, because any one of them would be out of date within weeks and would misrepresent how much the picture varies. The genuinely useful conclusions are structural and stable:

  • Training crawlers return almost nothing. They exist to harvest, not to refer.
  • Search and user-action bots do return traffic. They’re a different proposition entirely.
  • The spread between operators is enormous, so a blanket “allow AI bots” or “block AI bots” policy is a blunt instrument.
  • Your own ratio is calculable and is the only number that describes your site.

To compute yours: count AI bot requests by user agent in your logs over 30 days, then divide by the referral sessions your analytics attributes to each matching platform over the same window. The methodology for the referral side is in how to measure AI search traffic.

What should I actually look for in the log?

Look for five main metrics: which AI bots visit, how often they visit, which pages they fetch, which pages they ignore, and whether the pattern is changing over time. The gaps in activity are usually more informative than the hits.

  • Coverage. Are your most important commercial pages being fetched, or only your blog? If AI crawlers never reach your service pages, they can’t cite them.
  • Frequency. Independent log studies in 2026 reported GPTBot revisiting high-traffic pages every couple of days, with ClaudeBot and Google-Extended considerably slower. Your own cadence tells you how quickly content changes get picked up.
  • Status codes served to bots. A crawler receiving 404s or 500s is being told your content doesn’t exist.
  • Absence. No AI crawler hits at all usually means a robots.txt block, a CDN bot rule, or a site too new to be discovered.
  • Trend. One month is a snapshot. The direction of travel is the signal.

Do AI crawlers execute JavaScript?

Generally no. Multiple independent analyses in 2026 found that GPTBot, ClaudeBot and PerplexityBot fetch JavaScript files as text but do not execute them, so content that only appears after JS runs is effectively invisible to them. This is one of the most actionable findings available.

The practical consequence: if your pricing, your service descriptions or your key content render client-side, Googlebot may eventually see it, but AI crawlers largely won’t. A React or Vue site without server-side rendering can be fully indexed by Google and almost entirely opaque to ChatGPT.

How to check: fetch your own page with JavaScript disabled, or view the raw HTML source rather than the rendered DOM. If the content you want cited isn’t in the raw HTML, AI crawlers aren’t seeing it. The fix is server-side rendering, static generation, or at minimum making critical content available in the initial HTML response.

How do I know the bot was real?

Verify by IP, not by user-agent string. Anyone can send a request claiming to be GPTBot, and studies in 2026 estimated that roughly 5–8% of requests claiming to be known AI crawlers are spoofed. User agents are self-declared and trivially faked.

Verification approaches: the major operators publish IP ranges for their crawlers, and reverse DNS lookup on the requesting IP should resolve to the operator’s domain and forward-resolve back to the same IP. For most businesses, this matters less for analysis than for blocking decisions, but if you’re about to make a policy change based on heavy crawling from one operator, confirm it’s really them.

What do I do with what I find?

Three common outcomes and their fixes:

No AI crawler activity at all. Check robots.txt with our free checker, check for CDN or WAF bot rules, and confirm the site is discoverable. Being uncrawlable makes everything else in AI visibility moot.

Crawling but no citations. The crawlers can reach you but aren’t finding your content worth quoting. That points to a content structure and authority issue. See how to get cited by ChatGPT, Perplexity & Gemini and check whether AI systems can even identify your brand with the Entity Signal Checker.

Heavy training crawls, no referrals. This is the value-exchange question. You may decide to block training crawlers while allowing search and user-action bots, which keeps you citable without feeding models for free. Our llms.txt honest guide covers the control surfaces and which of them actually work.

Where this fits in OptiSEOn’s GEO work

Log-based crawl verification is part of how we approach LLM and AI search optimization. It’s the first diagnostic we run, because every other AI visibility tactic assumes crawl access that often isn’t there. Proving it with real data rather than estimates is the difference between a strategy and a hope.

Frequently Asked Questions

How do I know if GPTBot is crawling my site? 

Check your server or CDN logs for requests with the GPTBot user agent. This is the only reliable method, since analytics tools filter out bots and robots.txt only records your instructions. Upload a log to the free AI Crawler Log Analyzer to see activity by bot over time.

Why doesn’t Google Analytics show AI crawler traffic? 

Because analytics tools rely on JavaScript tags that fire in a browser, and crawlers generally don’t execute JavaScript. GA4 is designed to measure humans and actively filters known bots. Server logs capture every request regardless.

What is a good crawl-to-refer ratio? 

There’s no universal benchmark because it depends entirely on which operators crawl you and whether they run consumer-facing products that can send referrals. Published 2026 figures varied enormously by source and month. Calculate your own ratio per operator and track the direction over time rather than comparing to an industry average.

Do AI crawlers render JavaScript? 

Generally no. Independent analyses through 2026 found GPTBot, ClaudeBot and PerplexityBot fetch JavaScript files as text without executing them. Content that requires JavaScript to appear is effectively invisible to them, which makes server-side rendering important for AI visibility.

Should I block AI crawlers? 

It depends on the category. Training crawlers like GPTBot and ClaudeBot return little or no referral traffic; search and user-action bots like OAI-SearchBot, PerplexityBot and ChatGPT-User can send real visits. Many businesses allow the latter and make a deliberate choice about the former. Check your current rules with our robots.txt checker.

Can AI bot traffic be faked? 

Yes. User-agent strings are self-declared and trivially spoofed; 2026 studies estimated roughly 5–8% of requests claiming to be known AI crawlers were not genuine. Verify by IP range or reverse DNS before making policy decisions based on apparent crawl volume.


Stop guessing whether AI systems can see your site. Upload a server log and find out for free with no signup required. If the answer is “they’re not crawling you,” that’s the first thing to fix, and we can help.

Comments

One response to “Are AI Bots Actually Crawling Your Site? Your Server Logs Know”

  1. […] Content requiring JavaScript to render. Google can render JS, but it is slower and less reliable than plain HTML. AI crawlers also largely skip JS execution, as discussed in our AI crawler post. […]

Leave a Reply

Your email address will not be published. Required fields are marked *