robots.txt tester
Fetches a live robots.txt and checks whether a given path is allowed for a given crawler, using the same specificity rules real crawlers follow (most-specific user-agent group wins outright; within that group, the longest matching rule wins).
What is robots.txt tester?
robots.txt rules can get surprisingly subtle once multiple user-agent groups and overlapping Allow/Disallow rules are involved — the actual crawler behavior depends on specificity rules that aren't always obvious from just reading the file.
How to use it
Enter a domain, a path, and a crawler's user-agent — the tool fetches the live robots.txt and applies the same specificity rules real crawlers use (most-specific user-agent group wins outright; the longest matching rule wins within that group).
Example
Checking /admin/ for user-agent "*" against a robots.txt with
"Disallow: /admin/" -> Disallowed.