A reader interning at an online retailer wrote to me with a problem. Her manager had "blocked some URLs from indexing", yet pages generated by the catalogue filters kept appearing in Google. Were they bringing in any traffic? How do you make them go away? The question comes up constantly, and it exposes a common confusion: robots.txt does not govern indexing, it governs crawling. In a shop, where filters, sorts and pagination multiply addresses, that distinction changes everything.
Two things decide what a sound ecommerce robots.txt looks like: what the standard permits, and what Google actually does with it. Applied to a retail catalogue, that means faceted navigation, the quirks of WooCommerce, PrestaShop and Shopify, and the newer family of AI crawlers. The criteria for deciding what deserves blocking matter more than any example file — copying someone else's robots.txt is how shops lose category pages.
What robots.txt does, and what it does not
Robots.txt is a text file placed at the root of the site (your-shop.com/robots.txt). It tells crawlers which URLs they may request from the server. Since September 2022 its syntax has been described by RFC 9309, an IETF standard that Google, Bing, OpenAI and Anthropic all state they follow. It remains a code of conduct rather than a lock: a badly behaved crawler can ignore it.
The point Google's documentation repeats: robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still surface in results, without a description, if other pages link to it. That is exactly what my reader was seeing. To prevent indexing you need a meta robots noindex tag (or the X-Robots-Tag HTTP header), and Google can only read it if it is allowed to crawl the page. Blocking a page that is already indexed therefore freezes the situation instead of fixing it. One last reminder: the file is public. Never list a directory there that you want to keep private — use authentication. Separating the two mechanisms is the first step in almost any indexing problem.
Syntax worth knowing, and behaviour worth expecting
| Directive | Role | Note |
|---|---|---|
| User-agent | The crawler a group of rules applies to | * targets every crawler; a crawler obeys the most specific group that names it |
| Disallow | Forbids crawling of a path | Relative to the root; case-sensitive |
| Allow | Permits a path despite a broader Disallow | The longest rule wins; on a tie, Google applies the less restrictive one |
| Sitemap | Full URL of a sitemap | Several lines are allowed |
| Crawl-delay | Spacing between requests | Ignored by Google; documented by Bing |
Two special characters do most of the work in ecommerce: * stands for any sequence of characters, and $ marks the end of a URL. Disallow: /*?orderby= blocks every URL containing that parameter; Allow: /*?products=all$ permits only URLs ending exactly that way.
Google's documentation on how it interprets robots.txt spells out behaviours that are widely overlooked:
- Google reads at most 500 KiB; anything beyond that is ignored. The RFC sets the same minimum.
- The file is cached for up to 24 hours, so a correction does not take effect straight away.
- A 4xx error (404 included) is treated as no file at all: Google crawls everything. A 5xx error suspends crawling of the site for 12 hours, after which Google falls back on the cached version for up to 30 days. A robots.txt returning a server error is more dangerous than a missing one.
- Any directive other than user-agent, allow, disallow and sitemap is ignored.
Faceted navigation: what to block, and how
Filters by colour, size, price or brand, combined with sort options and pagination, generate thousands of URLs for a single catalogue. Google consolidated its advice in a dedicated page on faceted navigation (updated 18 December 2025). Its position: where those URLs are not meant to be indexed, the most effective approach is to prevent crawling through robots.txt, because crawling them consumes large amounts of resources. The alternatives — # fragments, canonical tags, rel="nofollow" on filter links — are presented as less reliable or slower to take effect. Google's own example transfers directly to a shop:
User-agent: *
Disallow: /*?*colour=
Disallow: /*?*size=
Disallow: /*?*price=
Disallow: /*?orderby=
Allow: /*?products=all$Three checks before you copy that block:
- A facet that matches a real search ("women's walking boots") deserves a clean URL, its own copy and indexing. Treat it as a category and give it the same content work you would give any landing page.
- Multiple filter combinations, sort orders and price bands add nothing to the index. Block them — but only after identifying the exact parameter names your platform uses.
- If you do leave some facets indexable, Google asks for a stable parameter order, the conventional
&separator, and a genuine 404 when a filter returns no products.
Keep crawl budget in proportion. Google states that managing crawl budget mainly concerns sites of one million unique pages or more, or of 10,000 pages or more whose content changes daily. A shop with 800 product lines is not in that bracket. The same page notes that a noindex tag saves no crawling: Google requests the page, then discards it. Only robots.txt prevents the request.
Removing filter URLs that are already indexed
Back to the original question: if filter URLs are already indexed, the order of operations matters. First check in Search Console, using the Performance report filtered to those URLs, whether they receive any clicks. If not, add a noindex tag and let Google keep crawling them long enough to drop them. Only then add the Disallow rule. Block first and the tag is never read, so the URLs stay visible for months. The Search Console removals tool hides a URL temporarily, but it does not replace this work.
Platform by platform: WooCommerce, PrestaShop, Shopify
Basket, checkout and account pages have no business in a search engine, and platforms already handle part of this. Check what yours does before adding rules.
WooCommerce (WordPress)
With no physical file present, WordPress serves a minimal virtual robots.txt: Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. Since version 5.0, WooCommerce adds a noindex directive to the Cart, Checkout and My account pages. Yoast SEO lets you create a real file (Yoast SEO, Tools, File editor). The parameters to watch: ?orderby= for sorting, ?filter_attribute= for filters, ?add-to-cart= and ?s= for internal search. Never block /wp-content/ or /wp-includes/ — Google needs the CSS and JavaScript files to render your pages.
PrestaShop
In PrestaShop 8, the "Generate robots.txt file" button sits under Shop Parameters, Traffic and SEO, in the SEO and URLs tab. The official documentation warns that the button overwrites the existing file: add your own rules after generating, and keep a copy. The generated file excludes technical directories for all crawlers; the parameters used by your filter module still have to be added by hand.
Shopify
Shopify generates a default robots.txt that blocks /cart, /checkout, /search, /admin and filtered collections. Customising it means creating a robots.txt.liquid template (Online Store, Edit code, Add a new template, type "robots"). Shopify's documentation classes this change as unsupported by its support team, warns that a mistake can cost you all your traffic, and recommends extending the rules through the Liquid objects rather than replacing them.
AI crawlers: a separate decision
A newer family of crawlers reads the web to train models or to feed generated answers. They identify themselves and document their behaviour, which makes it possible to treat them separately from search crawlers.
| User-agent | Operator | Declared purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search |
| ChatGPT-User | OpenAI | User-triggered actions; robots.txt may not apply |
| ClaudeBot | Anthropic | Collection for training |
| Claude-SearchBot, Claude-User | Anthropic | Search and on-demand retrieval for a user |
| Google-Extended | Training and grounding of Gemini models; no effect on Google Search |
The scale of this is measurable. In its Radar 2025 year in review (data to 2 December 2025, across the Cloudflare network), AI crawlers accounted for an average of 4.2% of HTML requests over the year, with GPTBot at roughly 7.5% of verified bot traffic. The same report notes that AI crawlers are the user-agents most often fully disallowed. An earlier Cloudflare analysis (1 July 2025) counted around 14% of the top 10,000 domains carrying directives aimed at these crawlers.
Should you block them? Training crawlers such as GPTBot, ClaudeBot and Google-Extended bring you no visitors, and blocking them has no effect on conventional search rankings — Google says so explicitly for Google-Extended. AI search crawlers such as OAI-SearchBot and Claude-SearchBot may cite your product pages in generated answers, so blocking those costs you visibility. Keep the two decisions apart:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /A complete example for a WooCommerce shop
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /*?*filter_
Disallow: /*?add-to-cart=
Disallow: /*?s=
Sitemap: https://www.your-shop.com/sitemap_index.xmlTreat this as a starting point, not a universal template: slugs, parameters and filter URLs depend on your theme and extensions. Before publishing, use the crawl stats report in Search Console to confirm that nothing useful matches the patterns you are blocking.
Testing and monitoring
The old Search Console robots.txt tester was retired at the end of 2023. Its replacement is the robots.txt report, available for domain-level properties: files found on the first 20 hosts, the date of the last read, parsing errors, and a way to request a re-read. The URL inspection tool tells you, page by page, whether crawling is permitted. Then keep an eye on the file after every theme or plugin update — that is usually how a staging Disallow: / ends up in production. This check belongs in a technical SEO audit, and it is worth scoping that audit around your real filter parameters rather than a generic checklist.
Common questions
Is robots.txt enough to remove a page from Google?
No. Robots.txt prevents crawling, not indexing. A blocked URL can stay in the results, without a description, if other pages link to it. Removing it requires a meta robots noindex tag Google can actually read — so the page must stay crawlable — or a removal request in Search Console.
Should an online shop block its filter pages?
Google recommends preventing the crawling of facet URLs that are not meant to be indexed, using Disallow rules aimed at the relevant parameters. Filters matching genuine searches are the exception: they deserve a clean URL and indexing, like a category page.
What happens if the robots.txt file returns an error?
Google treats a 404 as no file at all, so the whole site is crawlable. A 5xx server error suspends crawling for 12 hours, after which Google uses the cached version for up to 30 days. A server error is therefore more damaging than a missing file.
How do you edit robots.txt on Shopify?
Shopify generates the file automatically. To customise it you create a robots.txt.liquid template from the theme code editor. Shopify classes the change as unsupported and recommends extending the rules through the Liquid objects rather than replacing them.
Does blocking GPTBot or ClaudeBot harm SEO?
No. Those crawlers serve model training and are not connected to ranking in Google or Bing. Google states that the Google-Extended token has no effect on presence in Google Search. AI search crawlers such as OAI-SearchBot are a separate decision.
Where can you test a robots.txt now that the old tool is gone?
The Search Console robots.txt report shows the files Google has read, when, and any parsing errors, and lets you request a re-read. The URL inspection tool tells you whether a given page is crawlable.