Two tags, two different kinds of statement
rel="canonical" answers a question about identity: when the same content is reachable at several addresses, which one do you consider the real one. meta name="robots" answers a question about permission: may this page be listed, may its links be followed, may a snippet be shown. They live next to each other and get set together, which is exactly why they end up contradicting each other.
The crucial asymmetry is in how firmly each is taken. A robots directive is followed by the crawlers that honour robots directives at all. A canonical is a hint — weighed against internal linking, sitemaps, redirects and how similar the pages actually are, and overridable. Writing a canonical does not settle the matter; it registers a preference.
The four contradictions worth knowing
| Combination | Why it fails |
|---|---|
| noindex plus canonical to another URL | One says drop this page, the other says merge it into that page. The documented risk is the noindex carrying across to the target, so you lose the page you were trying to consolidate onto. |
| Canonical pointing at a URL that redirects | You have nominated an address that is not the final one. Follow the chain and name the destination. |
| Page 2 canonicalised to page 1 | Page 2 holds different items. Discarding it discards those items and anything only linked from them. Each paginated page should point at itself. |
| noindex on a URL blocked in robots.txt | Blocking prevents the fetch, so the tag is never read. The URL can still surface from external links, and now with no description at all. |
The last one is the most stubborn misunderstanding in this area, because both settings feel like they mean "keep this out" and stacking them feels safer. It is the opposite. Blocking a URL removes your ability to say anything about it.
When to use which
Same content at several addresses, all of which should stay reachable: canonical, pointing at the one you prefer. Content that has genuinely moved: a redirect, not a canonical. A page that should exist for visitors but not appear in search — an internal search results page, a thank-you page, a printable variant: noindex with a self-referencing canonical. A page nobody should reach at all: authentication, not tags.
The default for an ordinary page is a self-referencing canonical and no robots tag at all, since index and follow are what happens when you say nothing. Emitting index, follow explicitly is harmless and slightly self-documenting, which is why this tool writes it.
Parameters, case and slashes
Campaign parameters are the routine cause of one page becoming a hundred addresses. A canonical carrying utm_source is worse than none at all, since it enshrines one tagged variant as the official address. Strip tracking parameters from the canonical always. Sorting and filtering parameters need a decision instead: if the filtered view is genuinely different content, it can have its own self-canonical, and if it is the same items in another order, point it at the unfiltered URL.
Trailing slashes, uppercase letters and www versus bare domain all produce distinct URLs. Pick one form, enforce it with a server redirect, and make sure the canonical uses the form that survives the redirect rather than the one that triggers it. To see exactly what a messy URL decomposes into, the URL parser shows every component, and the UTM builder is where the campaign parameters should be added — to the link you publish, never to the canonical.
The header form
X-Robots-Tag does everything the meta tag does, sent as an HTTP response header, which means it works on files that have no head to put a tag in. A PDF, a generated spreadsheet, an image endpoint. It is also the only option when you cannot edit the template. Where both a header and a meta tag are present, they combine, and the more restrictive of the two wins — so a stray noindex in a server config will quietly override a well-intentioned index in your markup, and that is a genuinely painful afternoon of debugging if you do not know to look.
Related pieces: the robots.txt generator for crawl rules, which is a different mechanism from the directives here, the meta tag generator for the rest of the head, and the hreflang generator when the same content exists in several languages — a case where a careless cross-language canonical undoes the whole set.
Questions people ask
Should every page have a self-referencing canonical?
It is a reasonable default and costs nothing. It removes the ambiguity introduced by tracking parameters and stray uppercase, and it means a page that gets copied elsewhere carries a pointer home. The two cases where you would not: a page that genuinely is a duplicate of another, and a paginated series where you might reasonably decide the parameter handling is better done elsewhere.
Does a canonical stop duplicate content being a problem?
It expresses which address you prefer, and search engines mostly respect that. Whether "duplicate content" was ever a problem in the way it is usually described is a separate question — the plain effect is that several addresses for one page split your own links and your own analytics, which is reason enough to fix without invoking a penalty nobody outside a search engine can confirm.
Can I canonicalise to a page on another domain?
Yes, and it is the correct thing for syndicated content: the copy on the partner site names your original. It is also frequently a mistake, since a cross-domain canonical asks that the current URL not be listed at all. Check the hostname carefully after any migration; a canonical left pointing at the old domain is a common way to remove a whole site from an index.
What is the difference between blocking in robots.txt and noindex?
robots.txt controls fetching; noindex controls listing. A blocked URL is not fetched, so any tag on it is never read, and it can still appear in results as a bare URL discovered from links elsewhere. If the goal is that a page does not appear, it has to be crawlable so the noindex can be seen. If the goal is to stop the crawling itself, blocking is the tool and you give up the ability to say anything else about the page.
Do these tags work for AI crawlers?
Some honour robots directives, some publish their own user-agent names to target, and some do neither. Nothing here is enforcement — a directive is a request that well-behaved clients follow. If a page must not be read by anything you have not authorised, the answer is authentication, not a tag. The AI-specific user agents are covered on the robots.txt generator, and that list changes constantly.