# robots.txt tester

> Test whether a robots.txt allows Googlebot, GPTBot, ClaudeBot or any crawler to fetch a path. See the exact rule that decides it, using Google's matching.

Canonical: https://atrby.site/tools/robots-tester
Last updated: 2026-10-26

Open the interactive tool: [robots.txt tester](https://atrby.site/tools/robots-tester). It runs in the browser, with no sign-up.

## What it does

robots.txt rules interact in ways that aren't obvious. A broad Disallow and a narrow Allow can both match the same URL, and a crawler named in its own group ignores the rules for everyone else.

This tester answers one question at a time: may this crawler fetch this path? It shows the result and the line in your file that decided it.

## How to use it

1. Enter your domain under “Load robots.txt from a site (or paste it below)” and press “Load robots.txt”, or paste a draft straight into the “robots.txt” box.
2. Type a path or full URL into “Path or URL to test”, such as /blog/post.
3. Choose a “Crawler”: Googlebot, Bingbot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot or “Any crawler (*)”. For anything else, pick “Other…” and type its “User agent token”.
4. Read the result. It updates as you edit, so you can change a rule and see the effect straight away.

## How matching works

The tester follows Google's rules, which RFC 9309 also describes.

- A crawler uses the groups that name it. Only if none do does it use the “*” group.
- Among matching rules, the longest path wins.
- If an Allow and a Disallow are the same length, Allow wins.
- “*” matches any run of characters and “$” marks the end of the path.
- An empty Disallow line allows everything.

## Mistakes this catches

The classic one: you add a group for GPTBot with one Disallow line, and suddenly it no longer follows your “*” rules. Test it as GPTBot and as “Any crawler (*)” to see the difference.

Another: “Disallow: /blog” also blocks /blog-archive and /blogroll, because rules match by prefix. Use “/blog/” if you mean the folder.

Test drafts here before you upload them. A typo in robots.txt can block a whole site.

## Questions

### Is Google-Extended a real crawler?

Not in the usual sense. Googlebot does the fetching, and the Google-Extended token tells Google whether it may use the content for its AI models. Testing it shows what your rules say for that token.

### Are paths case-sensitive?

Yes. /Admin and /admin are different paths. User agent names are matched without regard to case.

### Does “Blocked” mean the page won't appear in Google?

Not always. Google can list a blocked URL without its content if other pages link to it. To keep a page out of results, allow crawling and add a noindex tag.

### Can I test query strings?

Yes. Paste a full URL and the tester keeps the path and query string, so rules like “Disallow: /*?sort=” can be checked.

## Related
- [All free tools](https://atrby.site/tools)
- [XML sitemap generator](https://atrby.site/tools/sitemap-generator)
- [Sitemap validator](https://atrby.site/tools/sitemap-extractor)