Robots.txt generator
Build a robots.txt from allow and disallow rules plus your sitemap URL.
FreeRuns in your browserNo signupUpdated 2026-10-29
robots.txt controls crawling, not indexing. Use a noindex tag to keep a page out of results. AI crawler names last checked October 2026.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Disallow: /search? Allow: /wp-admin/admin-ajax.php Sitemap: https://www.example.com/sitemap.xml
Upload it to the site root so it is served at https://www.example.com/robots.txt
Checks
- No problems found
About this tool
How it works and how to read the result.
Need this done for you?
Technical SEO & performance
Faster pages, clean structure and better visibility in search.
See the serviceWhat it does
A robots.txt generator writes the small text file that tells search engine crawlers which parts of your site to skip. You choose a default policy, list the paths to disallow, optionally re-allow specific files, add your sitemap URLs and decide whether to block AI training bots. The output is checked for common mistakes, such as paths that do not start with a slash, and can be copied or downloaded as a ready-to-upload file.
How to use it
- Enter the site URL so sitemap paths become absolute.
- Keep "Allow everything except the paths below" for a live site. Pick "Block the whole site" only for staging.
- List the folders and URL patterns to disallow, one per line, for example /admin/ or /search?.
- Add your sitemap path and tick the AI crawler box if you want those bots kept out.
- Download the file and upload it to the site root.
How to read the result
The output always begins with "User-agent: *", which applies to every crawler, followed by your rules. A typical small site ends up with four Disallow lines, one Allow line and one Sitemap line, about ten lines in total. If you have blocked /wp-admin/ but your theme loads admin-ajax.php from that folder, the Allow line re-opens that one file so the site still works for Googlebot.
The Checks panel lists anything that needs a second look. A path written as "admin/" without the leading slash will not match anything, so it is flagged. Crawl-delay is flagged because Google ignores it; if your server is struggling, fix the server rather than slowing crawlers. A missing sitemap is flagged because it is the cheapest way to help crawlers find new pages.
If you chose to block the whole site, a red badge reminds you that every crawler is locked out. That is correct for staging and wrong for anything you want found.
Test a specific URL against the finished file with the robots.txt tester, or ask about a full crawl audit through our technical SEO service.
Questions
Frequently asked.
Does Disallow stop a page from appearing in Google?
No. It stops Google from crawling the page, but the URL can still be indexed if other pages link to it, usually with no description. To keep a page out of results, let it be crawled and add a noindex meta tag or header. Use Disallow for crawl budget and private areas, not for hiding pages.
Where does the file go?
At the root of the host, so it is served at https://www.example.com/robots.txt. Each subdomain needs its own file. A robots.txt in a subfolder is ignored. After uploading, open the URL in a browser to confirm it loads as plain text.
Should I block AI crawlers?
It depends on whether you want your content used for model training. Blocking GPTBot, ClaudeBot and similar bots does not affect Google search rankings, and Google-Extended controls Gemini training without affecting Googlebot. Blocking them also removes you from some AI answer products, so decide with that trade-off in mind.
Your next move
Like what you see? Tell us about it.
A process that takes too long, a website that undersells you, or an idea ready to take shape. Tell us where you are and we will suggest a useful next step.
India · Working with clients worldwide