# Now Live: Crawler Analytics for AI Crawlers, Paths and Citations

- Canonical: https://limelit.co/blog/now-live-crawler-analytics-for-ai-crawlers-paths-and-citations
- Published: 2026-09-05
- Publisher: Limelit (https://limelit.co)

> Crawler analytics is live in Limelit: see which AI crawlers fetch your site, which paths they hit, where they fail, and which of those pages the engines then cite.

Crawler analytics is live in Limelit, under Site health. It shows which AI crawlers reach your site, which paths they fetch, where their requests fail, and, joined to the citation data Limelit already tracks, which of those pages the engines then cite in answers. It is the difference between "GPTBot visited" and "GPTBot fetched the pricing page 40 times and ChatGPT cited it twice."

## Why crawl logs alone were not the answer

Server logs already tell you a crawler came by. What they cannot tell you is whether the visit turned into anything. An engine can fetch a page every day and never cite it, or fetch it once and cite it for months. The join is the point: crawl on one side, citation and prompt topic on the other, one row per path.

The other gap is identification. A growing share of agent traffic does not identify itself. Coding agents in particular fetch documentation with plain HTTP clients, so a named crawler count understates what is really reading you. Crawler analytics shows the named crawlers and the paths, so you can see where the unnamed traffic concentrates.

## What it does

- **Crawlers.** Every known AI crawler that hit your site in the window, with visits, most-fetched paths and an intent filter: training, search, or user query.
- **Paths.** One row per URL: crawls, the crawlers that fetched it, citations, and the prompt topics those citations answered. Sort by any of them to find the pages engines read but never cite, and the pages they cite without recently reading.
- **Errors.** Requests that failed, by crawler and path, so a blocked or broken page is visible before it costs you a citation.
- **AI referrals.** Beside the crawlers, the sessions AI assistants actually sent you, from GA4, with engaged sessions and conversions.

## Connecting your logs

The setup wizard takes four kinds of source, and none of them requires an agent on your server:

1. A **Cloudflare Worker** that forwards AI crawler requests.
2. A **Vercel log drain**.
3. A **generic webhook** for any platform that can post request logs.
4. A **CSV or CLF upload** for a one-off look at an existing log.

Pick one, follow the two steps the wizard shows, and rows appear within minutes. The free [AI crawler check](https://limelit.co/tools/ai-crawler-check) is the zero-setup version: it reads your live robots.txt and tells you which crawlers you are letting in at all.

## What it costs

Nothing per row. Ingesting and joining logs does not use model calls, so it does not spend credits. The same data is available to your own agent through the MCP server: crawler visits, the crawl-to-citation join, crawl errors and the crawler policy.

## FAQ

**Which crawlers does it recognize?**
The known AI crawlers: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google's AI crawlers and the rest of the published list, each tagged with its intent. Unrecognized clients are still counted by path.

**Does it slow my site down?**
No. The Cloudflare Worker and the Vercel log drain forward request records after the response is served, and the upload path reads a file you already have.

**How does this relate to the AI readiness score?**
The readiness score says whether a page is easy for an engine to read and quote. Crawler analytics says whether engines are actually reading it and whether that turns into citations. Run both: readiness first, analytics to confirm.