# AI traffic and crawlers

> AI assistant visits get their own channel, AI crawler hits are reported from your server, and AI readiness checks robots.txt, llms.txt and the homepage.

Canonical: https://atrby.site/docs/crawlers
Last updated: 2026-10-21

People who arrive from AI assistants, and the AI bots that read your pages.

## Visits from AI assistants

Visits referred by an assistant get the **AI** channel, so you can see their visitors, revenue and conversion next to search and social. Nothing to set up.

| Assistant | Referrer hosts |
| --- | --- |
| ChatGPT | `chatgpt.com`, `chat.openai.com` |
| Perplexity | `perplexity.ai` |
| Claude | `claude.ai` |
| Gemini | `gemini.google.com` |
| Copilot | `copilot.microsoft.com` |
| Others | DeepSeek, Grok, You.com, Poe, Meta AI |

Links tagged `utm_source=chatgpt`, `perplexity` or `claude` count as AI too.

## AI crawlers

Crawlers like GPTBot and ClaudeBot don't run JavaScript, so the browser script never sees them. Report them from your server. Only the bot's user agent and the path are sent, and crawler hits are never billed as pageviews.

*Next.js · middleware.ts*

```ts
import { NextResponse, type NextRequest } from "next/server";

const AI_BOTS = /ChatGPT-User|OAI-SearchBot|GPTBot|Claude-User|Claude-SearchBot|ClaudeBot|Perplexity|Google-Extended|Applebot-Extended|meta-externalagent|Bytespider|CCBot|Amazonbot|DuckAssistBot|MistralAI-User/;

export function middleware(req: NextRequest) {
  const ua = req.headers.get("user-agent") ?? "";
  if (AI_BOTS.test(ua)) {
    fetch("https://app.atrby.site/api/ingest", {
      method: "POST",
      body: JSON.stringify({ batch: [{ type: "crawl", siteId: "st_your_site_key", ua, path: req.nextUrl.pathname }] }),
    }).catch(() => {});
  }
  return NextResponse.next();
}
```

Any server can do the same: post `{ batch: [{ type: "crawl", siteId, ua, path }] }` to `/api/ingest` when a request's user agent matches.

## AI readiness

AI traffic → Readiness checks your site from our server: it fetches `/robots.txt`, `/llms.txt` and the homepage on your domain only (5–8 second timeouts, size caps, at most 3 redirects on the same site) and keeps the result for an hour. It reports:

- What robots.txt lets each AI crawler do on the homepage: allowed, blocked, or allowed with some paths blocked, and whether the bot is named or follows the `*` rules.
- Whether llms.txt exists, its size, a structure score and whether up to 10 of its links on your domain answer.
- Whether the homepage has a title and meta description, and how long it took to answer once from our server.

We can't change files on your server. The Allow and Block switches build a recommended robots.txt that keeps your other rules; copy it and upload it yourself.

## What the intents mean

| Intent | Bots |
| --- | --- |
| Answer | Fetching a page right now to answer someone's question: ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User. |
| Index | Building an AI search index: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot. |
| Train | Collecting training data: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, CCBot. |

**Allow or block them.** Well-behaved crawlers follow robots.txt. Give each bot its own group, for example `User-agent: GPTBot` then `Disallow: /` to stop training while answer bots like ChatGPT-User stay allowed. The free [robots.txt generator](https://atrby.site/tools/robots-generator) gives you a starting file.

## Related
- [Goals and funnels](https://atrby.site/docs/goals-funnels)
- [Web vitals](https://atrby.site/docs/vitals)
- [Alerts and emails](https://atrby.site/docs/alerts)