# AI crawler access checker

> Enter a site and see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 12 other AI crawlers are allowed, partly blocked or blocked.

Canonical: https://atrby.site/tools/ai-crawler-access
Last updated: 2026-10-26

Open the interactive tool: [AI crawler access report](https://atrby.site/tools/ai-crawler-access). It runs in the browser, with no sign-up.

## What it does

Your robots.txt may already block AI crawlers you never thought about. Rules written years ago, a hosting default or a plugin can close the door on AI assistants without anyone noticing.

This report reads your live robots.txt and lists 16 AI crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance, Common Crawl, Amazon, DuckDuckGo and Mistral. For each one it shows whether it can reach your homepage and which rule it follows.

## How to use it

1. Type your domain or any page on it into “Website or page URL”. The report always reads robots.txt from the root of that domain.
2. Press “Check AI access”.
3. Read the three tables: AI assistants answering a question now, AI search indexes, and AI training data. Each row shows the crawler, its company, its access and its rule.
4. If something is wrong, open the robots.txt generator to write a corrected file.

## How to read the results

Each crawler gets one of three statuses. The “Rule” column tells you whether your file names that crawler directly or whether it falls back to the rules for all crawlers.

- allowed: it can fetch your homepage and no paths are closed to it.
- partial: it can fetch your homepage, but some paths are disallowed.
- blocked: it can't fetch your homepage at all.

## The pattern to look for

A common mistake is a blanket “Disallow: /” for all crawlers left over from a staging site. Another is blocking every AI crawler when the goal was to stop training. That second one also blocks the answer and search bots that let assistants cite you.

If you see answer or index bots marked blocked and that wasn't your intent, fix those first. They're the ones that can send people to your site.

If there's no robots.txt at all, the report says so. Every crawler is allowed by default.

## Questions

### What's the difference between answer, index and training bots?

Answer bots fetch a page when someone asks an assistant about it. Index bots build an AI search index ahead of time. Training bots collect data used to train models.

### Does “allowed” mean the crawler visits my site?

No. It only means your robots.txt doesn't stop it. Whether a crawler actually comes shows up in your server logs. atrby records AI crawler visits and the AI assistant referrals that turn into paying customers.

### Do all AI crawlers obey robots.txt?

robots.txt is advisory. The large AI companies publish their user agents and say their crawlers respect it. Some say fetches a person asks for may be treated differently, so check each company's crawler docs.

### Why does a crawler say “Follows the * rules”?

Your file doesn't mention it by name, so it uses the rules under “User-agent: *”. Name it in its own group if you want it treated differently.

## Related
- [All free tools](https://atrby.site/tools)
- [Favicon checker](https://atrby.site/tools/favicon-checker)
- [HTTPS redirect checker](https://atrby.site/tools/ssl-checker)