An online shop produces duplicate content by design. Every colour of a polo shirt can have its own URL, every filter combination creates another page, every category splits into numbered pages, and product descriptions often arrive unchanged from the supplier catalogue. None of that is a fault in itself. The problem starts when nobody tells Google which version counts, and the crawler spends its time on pages that sell nothing instead of the ones that do.
The question usually reaches me in the same shape. The owner of a menswear shop put it plainly: what is the real risk, and what are the good practices? His own diagnosis had three parts. Each colour or size sat on a different URL. Many product pages shared exactly the same description. The image alt attributes were empty.
Internal duplication is not a punishment, it is waste. It splits signals across several URLs, consumes crawl time and leaves Google to decide which page to show. What follows is the method applied to that catalogue, in the order it was applied.
What Google actually does with duplicate content
One misconception needs clearing first. Google Search Central documentation describes no duplicate content penalty. When several URLs carry the same content, Google picks one — the canonical — and consolidates the signals on it. You can steer that choice: redirects and the rel="canonical" link element are presented as strong signals, presence in the sitemap as a weak one, and these methods stack.
Two qualifications matter. Google's spam policies target scraped content — material copied from other sites without added value — and pages produced at scale with no purpose for the reader. A manufacturer description republished word for word by two hundred retailers sits close to that line. Second, crawl budget becomes a real subject as soon as a significant share of your URLs appears in Search Console as Discovered — currently not indexed. Google also names sites of more than 10,000 pages whose content changes daily, a threshold a shop with combinable filters reaches quickly.
The four sources of duplication in a shop
Product variants
Google accepts both architectures: one URL per variant, as a path segment (/mens-polo/blue) or a parameter (/mens-polo?colour=blue), or a single page with a selector. If variants have their own URLs, Google's ecommerce documentation asks you to place the canonical link of the main product on every variant page. With a single page, the base URL with no variant preselected should be the only canonical URL of the group, and each variant must stay reachable through a parameter. That accessibility is what makes ProductGroup structured data and Merchant Center feeds work, since those feeds group variants with the item_group_id attribute.
When a variant has demand of its own — people searching for "navy polo shirt men" rather than the product family — a separate page, self-canonical and genuinely differentiated, earns its place. Otherwise a canonical to the parent product is enough. The test: is there anything to say about this colour that you do not say about the others? If not, do not make it indexable.
Faceted navigation
Size, price, material and sort filters are the first source of useless URLs. Google's guidance on faceted navigation is blunt: in most cases there is no good reason to let filter combinations be crawled, and blocking them in robots.txt is the preferred method. Filtering handled through a URL fragment (#colour=blue) has no effect on crawling at all. The canonical link and the nofollow attribute are judged less effective in the long run here.
User-agent: *
Disallow: /*?colour=
Disallow: /*?size=
Disallow: /*?sort=
Disallow: /*&colour=Test any such rule against real URLs first: a slightly too broad pattern will block the category pages you want crawled. If some facets deserve indexing — a "blue polo shirts" page with genuine search volume — treat them as real category pages: the standard & separator between parameters, a filter order that never varies in the URL, their own introductory content, and a 404 for combinations that do not exist rather than a redirect.
Pagination
Pages 2, 3 and 4 of a category are not duplicates of page 1, and Google explicitly asks you not to canonicalise them to the first page: each paginated page gets its own canonical URL. The rel="next" and rel="prev" link elements are no longer used. What counts is a plain <a href> link to the next page and, ideally, one back to the first. The crawler does not click "load more" buttons: products loaded on demand must also be reachable through ordinary links or the sitemap.
Supplier descriptions
This is external duplication, and the only kind that touches the spam policies. Republishing the manufacturer's sheet puts you in competition with every other retailer and with the manufacturer itself, which has precedence. On a catalogue of several thousand references, rewriting everything is not realistic. Work in tiers: full rewrites for the products that make your revenue, a block of your own on the next tier — usage advice, care instructions, sizing, real customer questions — and the raw supplier sheet left unindexed for the long tail nobody searches for. The same logic governs your SEO content generally: volume without differentiation earns nothing.
| Source | Default treatment | Exception |
|---|---|---|
| Variants (colour, size) | Canonical to the parent product, or a single page with a selector | A dedicated page when the variant has its own demand and its own content |
| Facets and sorting | Robots.txt block, or filtering through a URL fragment | An indexable facet, treated as a category page in its own right |
| Pagination | A self-referencing canonical on each page, HTML links to the next | None: never canonicalise to page 1 |
| Supplier descriptions | Rewrite or enrich the priority products first | Noindex on the long tail that stays unrewritten |
Auditing the catalogue in five steps
- The Page indexing report in Search Console. Three reasons matter: Alternate page with proper canonical tag, the normal state of affairs; Duplicate without user-selected canonical, meaning you declared nothing; and Duplicate, Google chose different canonical than user, meaning your declaration is overruled.
- A full crawl. A crawler such as Screaming Frog or Sitebulb gives you URLs discovered against real products, duplicated titles and meta descriptions, missing or contradictory canonicals, and click depth. On the menswear site, the gap between crawled URLs and catalogue references showed the scale of the problem immediately.
- Crawl statistics. The Crawl stats report, or your server logs, shows where Googlebot actually spends its time. If parameter URLs dominate, your priority is robots.txt, not rewriting product copy.
- A sample of product pages. Take your twenty best sellers and search Google for an extract of each description in quotation marks. If ten sites return the same text, you have your rewriting list.
- Consistency of signals. The same URL in internal links, in the sitemap and in the canonical link element. Google repeats this rule for ecommerce sites, and it is the one plugins break most often.
This work belongs inside a wider technical SEO audit: category depth, page titles, internal linking, loading speed. On that menswear catalogue the home page title held only the brand name and the main categories were missing from the menu; both were fixed before anyone touched a canonical link. It helps to frame the audit before commissioning it.
Mistakes that cancel out your fixes
- Noindex on a URL blocked in robots.txt. Google cannot read a directive on a page it is not allowed to fetch, so the URL can stay indexed indefinitely. Choose one method, never both.
- Session or tracking identifiers in internal links. Every visit then creates fresh URLs. Google's advice is never to link internally to that kind of temporary parameter.
- Blocking facets after letting them index. Blocking access means those URLs can no longer be recrawled and dropped. Let the noindex take effect first, then block.
- Trusting the plugin defaults. A new filter module or review extension can reopen the leak on the day it is installed.
Keeping control of indexing is not a one-off operation. It is a check to repeat whenever the shop gains a feature.
What order to work in
Deal first with whatever produces the most URLs for the least value: facets and parameters, through robots.txt or fragments. Then the variant architecture. Then pagination. Content comes last, starting with the products that sell.
Expect several weeks between a fix and its reflection in the indexing report. Recrawling follows the rhythm of your site, not your deadline. Content rewriting, by contrast, is never finished: it belongs in the process of publishing every new product, not in a project that closes.
Common questions
Do I need a separate URL for each product colour?
No. Google accepts a single page with a selector just as readily as one URL per variant, provided every variant page carries a canonical link to the parent product. A dedicated page for one colour is justified only when that colour is searched for in its own right and has content of its own.
Does Google penalise duplicate content?
Google's documentation describes no penalty for internal duplication: Google selects a canonical version and consolidates signals on it. Content copied from other sites without added value is a different matter and falls under the spam policies. The internal risk is dilution and wasted crawl budget.
Can I keep the supplier's product descriptions?
You can keep them as a base, but dozens of retailers publish the same text and the manufacturer published it first. Rewrite or enrich the products that generate your sales, add a section of your own — usage, care, sizing — to the next tier, and set the untouched remainder to noindex.
Canonical or noindex for filter pages?
For filter combinations with no search demand, Google's preference is to prevent crawling in the first place, through robots.txt or URL fragments. The canonical link is less effective here in the long run. Noindex works, but only while the page stays reachable, which rules out combining it with a robots.txt block.
Should pages 2, 3 and 4 of a category canonicalise to page 1?
No. Google explicitly asks for each paginated page to have its own canonical URL. The rel="next" and rel="prev" link elements are no longer used. What matters is an ordinary HTML link to the following page and, where possible, a link back to the first one.
How do I know whether Google is ignoring my canonical tags?
The Page indexing report in Search Console lists the reason "Duplicate, Google chose different canonical than user". The URL Inspection tool shows, page by page, the canonical you declared and the one Google selected. A mismatch usually means internal links or a sitemap contradicting the tag.