Line Tools

Order of operations decides the answer here. Trimming happens before deduping, so "apple " and "apple" collapse into one line — leave trimming off and they stay two separate lines, which is occasionally what you want and usually a sign that the source data has a problem.

Line Tools — Remove Duplicate Lines, Sort, Trim and Number a ListBuildFigure

The steps run in a fixed order

Whatever you tick, the sequence is always the same: trim, then drop blanks, then dedupe, then sort, then reverse, then number. That order is not arbitrary and it changes the answer.

Trimming first means trailing whitespace stops hiding duplicates. A list copied out of a spreadsheet or a PDF frequently has a trailing space on some rows and not others, and until those are trimmed the deduper sees two different strings. Numbering last means the numbers reflect the final order, so if you sort and number in the same pass you get 1, 2, 3 down the sorted list rather than the original positions carried along. Reversing sits between sorting and numbering, which is how you get a descending list numbered 1 upward.

Three ways to sort, and when each is right

A to Z uses the browser's own locale comparison with numeric collation switched on. That gives you natural order, where file2 sorts before file10. A plain code-point sort — what most command-line tools do by default — puts file10 first because the character 1 comes before the character 2, and that is the single most common complaint about sorted file lists. Case is ignored for ordering, so Apple and apple sit together instead of all capitals coming first.

Sorting by the number in the line pulls the digits, minus sign and decimal point out of each line and compares those as a number, so $3,500 invoice sorts against $412 invoice correctly. It takes the digits in order of appearance, so a line with a date and an amount will sort by whichever comes first in the text. Lines with no digits at all count as zero and fall back to alphabetical order among themselves.

Shortest first is useful for spotting truncated rows and stray fragments at the top of a data dump, and for finding the outlier long line at the bottom.

Duplicates that do not look like duplicates

Deduping compares whole lines as exact strings, after trimming and after case folding if you asked for it. Two lines that look identical on screen can still be different strings, and the usual culprits are worth knowing:

What it looks likeWhat it actually isFix
Trailing spaces or tabsDifferent stringsTrimming handles it
A no-break space, U+00A0Not a space to the trimmerFind and replace it first
Curly quote vs straight quoteDifferent charactersNormalise the source
Full-width Latin lettersDifferent code pointsNormalise the source
e + combining accent vs éDifferent byte sequences, same glyphUnicode NFC normalisation

The last one is the hardest to see, because both render as an identical letter. If a list refuses to dedupe and you have ruled out whitespace, that is usually what is happening.

Rows with fields in them

Everything here compares and sorts the entire line, not a column within it. Paste CSV and two rows that differ only in a trailing timestamp are two distinct lines, which is correct behaviour and often not what you wanted. For column-aware work — dedupe on the second field, sort on the fourth — this is the wrong shape of tool, and you want a spreadsheet or a script. What it is good at is the one-item-per-line case: email lists, tag lists, URL lists, log lines, word lists, anything where the line is the record. Line endings are normalised on the way in, so CRLF and LF input both work and the output uses LF throughout.

Questions people ask

Does it handle Windows line endings?

Yes. CRLF and LF are both recognised on the way in and the result comes back with LF endings. That normalisation happens before anything else, so a file mixing both styles will not produce phantom differences or stray carriage returns hiding at the end of every line and defeating the deduper.

I can see duplicates but they are not being removed.

Turn trimming on first and try again — trailing whitespace is the cause about half the time. If they still survive, the lines differ in a character you cannot see. No-break spaces pasted from a web page, curly versus straight apostrophes, and precomposed versus decomposed accented letters all look the same on screen and compare as different strings. Case is also significant unless you tick the case-insensitive option.

How many lines can it handle?

It refuses above 200,000 lines and tells you, rather than locking the tab. Below that the limit is your browser: a few tens of thousands of lines is instant, and very large inputs may pause visibly while the sort runs because it all happens in one synchronous pass on the page. Nothing is sent anywhere, so pasting an internal list carries no upload risk.

Which duplicate is kept, the first or the last?

The first occurrence stays and later ones are dropped. If the position matters — you want the most recent of a set of repeated log lines rather than the earliest — reverse the list first in your source, or run the tool, then reverse the output. When case-insensitive matching merges Apple and apple, the survivor keeps the capitalisation of whichever appeared first.

Related