Robots.txt: what to block, what not to, and common mistakes
Robots.txt controls crawling, not indexing. What a business site should block, what it must never block, and how Google reads the file as of October 2026.
By Sohail Hussain21 November 20265 min read
Robots.txt is a plain text file at the root of your domain that tells crawlers which URLs they may fetch. For most business websites the right file is short: allow everything, block a few low-value URL patterns such as the admin area, internal search results and cart pages, and list your sitemap. Never use it to hide a page from Google. Robots.txt controls crawling, not indexing; to keep a page out of search, use noindex or a password.
A wrong robots.txt is one of the few single-line mistakes that can remove a whole site from search, which is why it is the first item in our technical SEO checklist and one of the first files we open in a technical SEO and performance audit.
What does robots.txt actually control?
Crawling. Google's introduction to robots.txt says the file is mainly for managing crawler traffic and "is not a mechanism for keeping a web page out of Google".
That distinction causes the most common mistake. A URL blocked in robots.txt can still appear in search results if other pages link to it. Google shows the URL, often with no description, because it was never allowed to read the page. Search Console flags this as "Indexed, though blocked by robots.txt".
It also means robots.txt and noindex work against each other. A noindex tag only works if Google can crawl the page and read it. Block the page in robots.txt and the tag is never seen, so the URL can stay indexed. To remove a page from search: allow crawling, add noindex, wait for Google to recrawl, and only then block it if you still want to.
What should a business site block?
| URL type | Block in robots.txt? | Why |
|---|---|---|
Admin and login (/wp-admin/, /admin) | Yes | No search value; WordPress needs admin-ajax.php left open |
Internal search results (/?s=, /search) | Yes | A new URL for every query anyone types, none worth crawling |
| Cart, checkout, account | Yes | Personal and empty for crawlers |
| Sort and filter parameters on large catalogues | Usually | Google's crawl budget guide names "differently sorted versions of the same page" as a case for blocking |
| Thank-you and confirmation pages | No; use noindex | Blocking does not keep them out of the index |
| Staging sites | No; use a password | Anyone can read robots.txt, and blocked pages can still be indexed |
| CSS, JavaScript, images, fonts | Never | Google needs them to render the page |
A typical file for a WordPress business site:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://www.example.com/sitemap_index.xml
And the extra lines a shop with filtered category pages might add:
Disallow: /cart
Disallow: /checkout
Disallow: /my-account
Disallow: /*?*sort=
Disallow: /*?*filter_
Check the exact paths your platform uses before copying either. Our robots.txt generator builds a file from your rules and sitemap URL.
What must you never block?
- The whole site.
Disallow: /underUser-agent: *blocks everything. It is common on staging sites and gets launched with the site more often than anyone admits. - Rendering resources. CSS, JavaScript and image folders. Google renders pages like a browser; without these files it sees a broken page.
- Pages you are redirecting. Google has to crawl the old URL to see the 301. Block it and the redirect is never followed.
- Pages carrying
noindexthat you want removed. As above, the tag must be readable. - Canonical targets and pages in your sitemap. Blocking a URL you have asked Google to index sends contradictory signals.
How does Google read the file?
The rules are in Google's robots.txt specification. The ones that matter in practice, as of October 2026:
- Location. The file must be at the root of each protocol and host:
https://www.example.com/robots.txtcovers only that host. A subdomain needs its own file. - Size. Google reads the first 500 KiB and ignores the rest.
- Caching. Google generally caches the file for up to 24 hours, so changes are not instant.
- Matching. Paths are case-sensitive.
*matches any characters and$marks the end of the URL. - Precedence. The most specific rule wins, meaning the one with the longest matching path. If an
Allowand aDisalloware equally specific, Google uses the less restrictive one. - Groups. A crawler follows only the most specific
User-agentgroup that matches it. If you add aUser-agent: Googlebotgroup, Googlebot ignores the*group entirely, so repeat any shared rules. - Supported fields.
user-agent,allow,disallowandsitemap. Google ignorescrawl-delay, andnoindexin robots.txt has no effect. - Errors. A 404 or other 4xx (except 429) means no restrictions. A 5xx makes Google stop crawling the site for up to 12 hours while it retries, then fall back to the last good copy for up to 30 days.
The group rule causes the most surprises. A file with a general * group and a separate Googlebot group that only allows one folder means Googlebot follows only the latter.
Should you block AI crawlers?
That depends on which crawler and what you want to stop.
- Googlebot crawls for Google Search, including AI Overviews and AI Mode. Blocking it removes you from Google Search entirely.
- Google-Extended is a separate token you can disallow in robots.txt. Google's AI features guide says it limits AI training and grounding in some of Google's other systems. It does not remove your pages from Google Search.
- To leave AI Overviews and AI Mode while staying in Search, robots.txt is the wrong tool. Search Console now has a Search generative AI control under Settings for that.
- Other companies' AI crawlers each publish their own user-agent names. Block them by name if you do not want your content used by those services, after checking their documentation for what the block covers.
For most business sites whose goal is enquiries, being cited in AI answers is more useful than being absent from them. AI Overviews and your traffic goes through that trade-off.
How do you test a change before it goes live?
- Test URLs against the new file. Paste the draft file into our robots.txt tester and check your most important URLs one by one: homepage, main service pages, a blog post, a CSS file. None should be blocked.
- Publish, then check the robots.txt report in Search Console (under Settings). It shows the version Google last fetched, when, and any parsing errors, for up to 20 hosts in a Domain property.
- Request a recrawl of the file from the same report if you have unblocked something important. Google otherwise picks up changes within about a day.
- Inspect a few key URLs with URL Inspection to confirm crawling is allowed.
If pages are already missing from search, why your pages are not indexed covers "Blocked by robots.txt" alongside the other reasons. And because the Sitemap: line belongs in this file, XML sitemaps: what should be in yours is the natural next read.
Was this guide useful?