Robots.txt: what to block, what not to, and common mistakes

Robots.txt controls crawling, not indexing. What a business site should block, what it must never block, and how Google reads the file as of October 2026.

By Sohail Hussain21 November 20265 min read

Robots.txt is a plain text file at the root of your domain that tells crawlers which URLs they may fetch. For most business websites the right file is short: allow everything, block a few low-value URL patterns such as the admin area, internal search results and cart pages, and list your sitemap. Never use it to hide a page from Google. Robots.txt controls crawling, not indexing; to keep a page out of search, use noindex or a password.

A wrong robots.txt is one of the few single-line mistakes that can remove a whole site from search, which is why it is the first item in our technical SEO checklist and one of the first files we open in a technical SEO and performance audit.

What does robots.txt actually control?

Crawling. Google's introduction to robots.txt says the file is mainly for managing crawler traffic and "is not a mechanism for keeping a web page out of Google".

That distinction causes the most common mistake. A URL blocked in robots.txt can still appear in search results if other pages link to it. Google shows the URL, often with no description, because it was never allowed to read the page. Search Console flags this as "Indexed, though blocked by robots.txt".

It also means robots.txt and noindex work against each other. A noindex tag only works if Google can crawl the page and read it. Block the page in robots.txt and the tag is never seen, so the URL can stay indexed. To remove a page from search: allow crawling, add noindex, wait for Google to recrawl, and only then block it if you still want to.

What should a business site block?

URL typeBlock in robots.txt?Why
Admin and login (/wp-admin/, /admin)YesNo search value; WordPress needs admin-ajax.php left open
Internal search results (/?s=, /search)YesA new URL for every query anyone types, none worth crawling
Cart, checkout, accountYesPersonal and empty for crawlers
Sort and filter parameters on large cataloguesUsuallyGoogle's crawl budget guide names "differently sorted versions of the same page" as a case for blocking
Thank-you and confirmation pagesNo; use noindexBlocking does not keep them out of the index
Staging sitesNo; use a passwordAnyone can read robots.txt, and blocked pages can still be indexed
CSS, JavaScript, images, fontsNeverGoogle needs them to render the page

A typical file for a WordPress business site:

txt
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://www.example.com/sitemap_index.xml

And the extra lines a shop with filtered category pages might add:

txt
Disallow: /cart
Disallow: /checkout
Disallow: /my-account
Disallow: /*?*sort=
Disallow: /*?*filter_

Check the exact paths your platform uses before copying either. Our robots.txt generator builds a file from your rules and sitemap URL.

What must you never block?

  • The whole site. Disallow: / under User-agent: * blocks everything. It is common on staging sites and gets launched with the site more often than anyone admits.
  • Rendering resources. CSS, JavaScript and image folders. Google renders pages like a browser; without these files it sees a broken page.
  • Pages you are redirecting. Google has to crawl the old URL to see the 301. Block it and the redirect is never followed.
  • Pages carrying noindex that you want removed. As above, the tag must be readable.
  • Canonical targets and pages in your sitemap. Blocking a URL you have asked Google to index sends contradictory signals.

How does Google read the file?

The rules are in Google's robots.txt specification. The ones that matter in practice, as of October 2026:

  • Location. The file must be at the root of each protocol and host: https://www.example.com/robots.txt covers only that host. A subdomain needs its own file.
  • Size. Google reads the first 500 KiB and ignores the rest.
  • Caching. Google generally caches the file for up to 24 hours, so changes are not instant.
  • Matching. Paths are case-sensitive. * matches any characters and $ marks the end of the URL.
  • Precedence. The most specific rule wins, meaning the one with the longest matching path. If an Allow and a Disallow are equally specific, Google uses the less restrictive one.
  • Groups. A crawler follows only the most specific User-agent group that matches it. If you add a User-agent: Googlebot group, Googlebot ignores the * group entirely, so repeat any shared rules.
  • Supported fields. user-agent, allow, disallow and sitemap. Google ignores crawl-delay, and noindex in robots.txt has no effect.
  • Errors. A 404 or other 4xx (except 429) means no restrictions. A 5xx makes Google stop crawling the site for up to 12 hours while it retries, then fall back to the last good copy for up to 30 days.

The group rule causes the most surprises. A file with a general * group and a separate Googlebot group that only allows one folder means Googlebot follows only the latter.

Should you block AI crawlers?

That depends on which crawler and what you want to stop.

  • Googlebot crawls for Google Search, including AI Overviews and AI Mode. Blocking it removes you from Google Search entirely.
  • Google-Extended is a separate token you can disallow in robots.txt. Google's AI features guide says it limits AI training and grounding in some of Google's other systems. It does not remove your pages from Google Search.
  • To leave AI Overviews and AI Mode while staying in Search, robots.txt is the wrong tool. Search Console now has a Search generative AI control under Settings for that.
  • Other companies' AI crawlers each publish their own user-agent names. Block them by name if you do not want your content used by those services, after checking their documentation for what the block covers.

For most business sites whose goal is enquiries, being cited in AI answers is more useful than being absent from them. AI Overviews and your traffic goes through that trade-off.

How do you test a change before it goes live?

  1. Test URLs against the new file. Paste the draft file into our robots.txt tester and check your most important URLs one by one: homepage, main service pages, a blog post, a CSS file. None should be blocked.
  2. Publish, then check the robots.txt report in Search Console (under Settings). It shows the version Google last fetched, when, and any parsing errors, for up to 20 hosts in a Domain property.
  3. Request a recrawl of the file from the same report if you have unblocked something important. Google otherwise picks up changes within about a day.
  4. Inspect a few key URLs with URL Inspection to confirm crawling is allowed.

If pages are already missing from search, why your pages are not indexed covers "Blocked by robots.txt" alongside the other reasons. And because the Sitemap: line belongs in this file, XML sitemaps: what should be in yours is the natural next read.

Was this guide useful?

Questions

Frequently asked.

Does robots.txt stop a page appearing in Google?

No. It stops Google crawling the page, but the URL can still be indexed from links on other sites, shown without a description. To keep a page out of search, allow crawling and add a noindex meta tag or header, or put the page behind a password.

Do I need a robots.txt file at all?

No. If the file is missing and the server returns a 404, Google assumes there are no restrictions and crawls everything it can find. Most business sites still benefit from a short file that blocks admin and search pages and lists the sitemap.

Can I use crawl-delay to slow Googlebot down?

No. Google ignores crawl-delay. If Googlebot is overloading the server, Google's crawlers slow down on their own when they receive 500, 503 or 429 responses. Fix the server capacity or caching rather than trying to throttle Google through robots.txt.

Sohail Hussain

Founder, MSH Technologies

Sohail has designed and built websites, web apps and AI products for over 12 years, for clients from Singapore-based industrial groups to early-stage SaaS startups. He writes about what actually worked.

Keep reading

Related guides.

All posts

Your next move

Like what you see? Tell us about it.

A process that takes too long, a website that undersells you, or an idea ready to take shape. Tell us where you are and we will suggest a useful next step.

India · Working with clients worldwide

Sohail, Founder, replies within 1 working day (Mon–Sat, IST). Free and no obligation.