Robots.txt tester
Paste a robots.txt and a URL to see whether a bot may crawl it.
FreeRuns in your browserNo signupUpdated 2026-10-29
Follows Google's rules: the most specific user-agent group applies, the longest matching path wins, and Allow beats Disallow on a tie.
Group applied
User-agent: *
2 groups found in the file
Blocked
3 of 6
URLs this user-agent may not crawl
| Path | Result | Matched rule |
|---|---|---|
| / | Allowed | No rule matched: allowed |
| /admin/users | Blocked | Disallow: /admin/ |
| /admin/public/logo.png | Allowed | Allow: /admin/public/ |
| /search?q=shoes | Blocked | Disallow: /search? |
| /brochure.pdf | Blocked | Disallow: /*.pdf$ |
| /private-images/team.jpg | Allowed | No rule matched: allowed |
About this tool
How it works and how to read the result.
Need this done for you?
Technical SEO & performance
Faster pages, clean structure and better visibility in search.
See the serviceWhat it does
A robots.txt tester checks whether a given crawler may fetch a given URL under your rules, without waiting for Search Console to tell you. Paste the file, choose a user-agent and list the URLs you care about. For each one you see allowed or blocked plus the exact rule that decided it. The matching follows the rules Google publishes: most specific user-agent group, longest matching pattern, Allow on a tie, with asterisk and dollar wildcards.
How to use it
- Paste the full contents of your live or draft robots.txt.
- Type the user-agent you want to test as, for example Googlebot or GPTBot.
- List the URLs to check, one per line. Full URLs or bare paths both work.
- Read the table. Each row shows the path, the verdict and the rule that produced it.
- Edit the file in the left box and watch the verdicts update.
How to read the result
The "Group applied" stat tells you which block of the file the chosen crawler uses. Testing as Googlebot against the sample file gives "User-agent: *", because no group names Googlebot directly. Switch to Googlebot-Image and the group changes to its own block, which means the admin and PDF rules no longer apply to it. That surprises most people and is the usual cause of "but I blocked that" confusion.
In the table, /admin/users is blocked by "Disallow: /admin/" while /admin/public/logo.png is allowed by "Allow: /admin/public/" because its pattern is 14 characters long against 7. The /brochure.pdf row is blocked by "/*.pdf$", showing both wildcards at work. A row that says "No rule matched: allowed" means the crawler is free to fetch it.
The Blocked stat gives a quick count. If it is higher than you expected, check for a Disallow with a short pattern such as "/" or "/p" that catches more than intended.
Build a clean file first with the robots.txt generator. For crawl problems that go beyond one file, see our technical SEO service.
Questions
Frequently asked.
Which rule wins when Allow and Disallow both match?
The one with the longest path pattern, because it is more specific. If both patterns are the same length, Allow wins. So "Disallow: /admin/" with "Allow: /admin/public/" lets the public folder through, since the Allow pattern is longer. Bing follows the same logic; some smaller crawlers only use the first match.
Why did my Googlebot-Image rule not apply to Googlebot?
Each crawler uses only the single most specific group that names it. Googlebot-Image has its own group, so it ignores the rules under "User-agent: *". Plain Googlebot does not match "Googlebot-Image", so it falls back to the star group. Rules are not merged across groups.
Do wildcards work in robots.txt?
Google and Bing support two. An asterisk matches any run of characters, and a dollar sign at the end anchors the match to the end of the URL. "/*.pdf$" blocks every URL ending in .pdf. Patterns are case-sensitive and always match from the start of the path.
Your next move
Like what you see? Tell us about it.
A process that takes too long, a website that undersells you, or an idea ready to take shape. Tell us where you are and we will suggest a useful next step.
India · Working with clients worldwide