Remove Duplicate Lines

What counts as a duplicate

By default two lines are duplicates only when they match exactly, with one deliberate exception: spaces at the start and end of a line are ignored, because they are invisible and almost never meaningful. “apple” and “apple ” are the same entry to anyone reading the list, and a tool that keeps both looks broken. Turn that option off when leading spaces matter, as they do in indented code or fixed-width data.

Ignore case makes “Apple”, “apple” and “APPLE” one entry, which is usually right for email addresses, tags and names typed by different people. It follows Unicode's case rules rather than only A to Z, so “STRASSE” and “straße” match as well.

Characters that Unicode defines as the same are compared as the same: an “é” stored as one character and an “é” stored as an “e” plus an accent count as one line. What no comparison can see is a difference that is invisible but real — a zero-width space pasted from a web page, a non-breaking space inside a line, curly quotes in one copy and straight ones in the other. If two lines look identical but both survive, the invisible character detector on this site will show you why.

Keeping the first copy or the last

Unlike sorting tools that remove duplicates as a side effect, this keeps your list in its original order. Keeping the first occurrence leaves each entry where it first appeared, which is what you want for a mailing list, a queue of URLs to process, or anything where the order carried meaning.

Keeping the last occurrence places each entry where it last appeared. That matters when later lines supersede earlier ones: a log in which the latest status of each item is the one that counts, a list of settings where the final value wins, or a list that has been appended to over months and should reflect each entry's most recent position.

A small example makes the difference clear. From apple, banana, apple, cherry, banana, keeping the first copy gives apple, banana, cherry; keeping the last gives apple, cherry, banana. Either way nothing is reordered except by the removals themselves.

Blank lines

Empty lines get their own setting because no single behaviour suits every list. Leaving them where they are preserves any structure the spacing provides — groups of addresses separated by blank lines, say — and never treats one blank line as a copy of another. Treating them as duplicates keeps only the first blank line and removes the rest. Removing them gives a tight list with exactly one entry per line, which is usually what a spreadsheet import or a script wants.

A line that contains nothing but spaces or tabs counts as empty. The result keeps the line endings it was given, so a file with Windows (CRLF) line breaks comes back with Windows line breaks, and a file that ended with a line break still does.

Large lists, privacy, and other ways to do it

Each line is checked against a hash table of the lines already seen, so the work grows in proportion to the length of the list rather than with its square. A list of 100,000 lines is processed in well under a second on an ordinary laptop; the practical limit is your device's memory, not the tool.

The lists people deduplicate are often personal data — customer emails, phone numbers, account IDs — and pasting them into a website that sends them to a server is a data-protection decision under GDPR and similar laws, whether or not anyone thinks of it that way. Here the list is processed in your browser and never transmitted.

If you would rather use a desktop tool: in Excel, Data → Remove Duplicates does the same job on a column and ignores case when comparing. On the command line, sort -u removes duplicates but sorts the list as it does so, while awk '!seen[$0]++' keeps the first occurrence of each line in the original order.