Five rules, and they do not produce the same list
Keeping the first occurrence is the default and the right answer for most contact lists, SKU lists and address books: the order of the file is preserved and every second-and-later copy disappears. Keeping the last occurrence matters when the file is append-only and later rows are corrections — an export where someone's address was updated will have the good record at the bottom.
Collapsing consecutive repeats only compares each line with the one immediately above it. That is what you want on a log file where the same message fired four hundred times in a row, and on any list that is already sorted and whose order you do not want disturbed. It will leave duplicates that sit apart from each other, and the page says so when that happens rather than letting you assume the list is clean.
The last two rules do not remove anything, they select. Showing only the repeated lines answers "what came in twice", which is usually the question when you are auditing an import. Keeping only lines that appeared exactly once answers the opposite question and is how you find the row that exists in one file and not the other.
What counts as the same line
Comparison is whole-line and exact, after whatever normalisation you switched on. The whitespace option is on by default because it has to be: a list assembled from a spreadsheet, a PDF and a web page will have trailing spaces on some rows and not others, and until those are stripped the comparison sees two different strings. Switching it on trims both ends and collapses runs of internal spaces, but only for the comparison — the line that survives into the result is your original text, spaces and all.
Case matching is off by default, because it is right for email addresses and wrong for identifiers, product codes and anything case-sensitive downstream. When you do switch it on, the surviving row keeps the capitalisation of whichever copy your rule selected, so the output can be internally inconsistent. Run it through the case converter afterwards if the case has to be uniform.
Duplicates that refuse to collapse
If two lines look identical on screen and still survive, the difference is a character you cannot see.
| What you see | What is actually there | What fixes it |
|---|---|---|
| Trailing space or tab | Two different strings | The whitespace option here |
| A no-break space, U+00A0 | Not whitespace to the trimmer | Replace it first |
| Curly vs straight apostrophe | Different code points | Normalise the source |
| Full-width Latin letters | Different code points | Unicode NFKC normalisation |
| e + combining acute vs precomposed é | Same glyph, different bytes | Unicode NFC normalisation |
The last row is the one that wastes an afternoon, because both forms render as an identical letter at any zoom level. Sorting A to Z will usually park the two variants next to each other, which at least makes the problem visible.
Non-Latin text, and where the line boundary is
Comparison is by code point, so any script works: Greek, Cyrillic, Arabic, Hebrew, Han, kana and Hangul lines dedupe exactly as Latin ones do, and case folding simply has no effect on scripts without a case distinction. Sorting A to Z uses the browser's English collation with numeric ordering switched on, which puts file2 before file10; for non-Latin text that collation still produces a stable, repeatable order, but it is not the order a native speaker of that language would call alphabetical. If the order matters more than the deduping, sort in a tool that knows the locale.
The unit of comparison is always the whole line. A row of comma-separated values is one item, not several, and two CSV rows that differ only in a trailing timestamp are two distinct lines. Split the data to one record per line before pasting it. The cap is 20,000 lines; past that the tool reads the first 20,000 and says so, because comparing and rendering more than that in one synchronous pass is where a browser tab stops responding.
Questions people ask
Does it keep my original order?
Yes, unless you ask it not to. With the order set to input order and the rule set to keep the first occurrence, the surviving lines come back in exactly the sequence you pasted them, with later copies removed. Keeping the last occurrence also preserves relative order — it changes which copy survives, not where the survivors sit. Only the three sort options reorder anything.
Which copy survives when I ignore case?
Whichever one your rule selects. Keeping the first occurrence keeps the capitalisation of the earliest copy, keeping the last keeps the final one. Because of that, a case-insensitive pass can leave you with a list where some entries are capitalised and some are not — the tool warns about this. If you need consistent casing, dedupe first and re-case the result afterwards.
Can I match on part of a line instead of the whole thing?
No. Comparison is always the entire line, and there is no substring, prefix or pattern matching. If your lines carry a trailing note or an ID that should not participate in the comparison, strip it first with a batch replace or the regex-capable multi find and replace, dedupe the normalised version, and rejoin. Approximate matching — treating "Acme Ltd" and "Acme Limited" as one company — is a judgement call no line comparison can make.
How large a list can it take?
Twenty thousand lines. Above that it processes the first 20,000 and tells you the input was truncated rather than freezing while it works. Everything happens in the page on your own machine, so nothing is uploaded and an internal customer list carries no transmission risk — but that also means the ceiling is your browser rather than a server, and very long lines can slow it down well before the line count does.