What it does
robots.txt rules interact in ways that aren't obvious. A broad Disallow and a narrow Allow can both match the same URL, and a crawler named in its own group ignores the rules for everyone else.
This tester answers one question at a time: may this crawler fetch this path? It shows the result and the line in your file that decided it.
How to use it
- 1Enter your domain under “Load robots.txt from a site (or paste it below)” and press “Load robots.txt”, or paste a draft straight into the “robots.txt” box.
- 2Type a path or full URL into “Path or URL to test”, such as /blog/post.
- 3Choose a “Crawler”: Googlebot, Bingbot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot or “Any crawler (*)”. For anything else, pick “Other…” and type its “User agent token”.
- 4Read the result. It updates as you edit, so you can change a rule and see the effect straight away.
How matching works
The tester follows Google's rules, which RFC 9309 also describes.
- A crawler uses the groups that name it. Only if none do does it use the “*” group.
- Among matching rules, the longest path wins.
- If an Allow and a Disallow are the same length, Allow wins.
- “*” matches any run of characters and “$” marks the end of the path.
- An empty Disallow line allows everything.
Mistakes this catches
The classic one: you add a group for GPTBot with one Disallow line, and suddenly it no longer follows your “*” rules. Test it as GPTBot and as “Any crawler (*)” to see the difference.
Another: “Disallow: /blog” also blocks /blog-archive and /blogroll, because rules match by prefix. Use “/blog/” if you mean the folder.
Test drafts here before you upload them. A typo in robots.txt can block a whole site.
Questions
Is Google-Extended a real crawler?
Not in the usual sense. Googlebot does the fetching, and the Google-Extended token tells Google whether it may use the content for its AI models. Testing it shows what your rules say for that token.
Are paths case-sensitive?
Yes. /Admin and /admin are different paths. User agent names are matched without regard to case.
Does “Blocked” mean the page won't appear in Google?
Not always. Google can list a blocked URL without its content if other pages link to it. To keep a page out of results, allow crawling and add a noindex tag.
Can I test query strings?
Yes. Paste a full URL and the tester keeps the path and query string, so rules like “Disallow: /*?sort=” can be checked.