Remove duplicate lines from a list

Clean up a list as soon as you paste it

This tool lets you remove duplicate lines from any list. Paste it under “Your list” and the “Result” box next to it shows the same list with each item kept once. It updates as you type or change an option.

Typical jobs: checking a mailing list before a newsletter goes out, so nobody gets the same email twice; deduplicating keywords for a campaign; tidying product codes, a guest list or a class register; cleaning a column copied from a spreadsheet; merging lists that came from several sources. The list stays in your browser: nothing is uploaded.

How to use it

1. Paste your list, one item per line. If the items are separated another way, pick “Comma”, “Semicolon” or “Tab” under “List separator”.
2. Keep or untick the four boxes: “Remove duplicates”, “Ignore case”, “Ignore leading and trailing spaces”, “Remove empty lines”.
3. Under “Sort”, keep “Original order” or choose “A → Z”, “Z → A”, “By length”, “Numeric order”, “Reversed order” or “Shuffled at random”.
4. Check the three counters: lines in, lines out, duplicates removed.
5. Click “Copy result”. “Clear all” empties both boxes.

Before and after: a worked example

Take six lines of email addresses, the fifth one empty.

The counters read 6 lines in, 3 lines out and 2 duplicates removed. Click “Show duplicates only” and the box lists what was repeated, with a count: info@example.com × 2 and Sales@example.com × 2.

What counts as a duplicate?

That depends on the boxes. With the starting settings, “London” and “london” followed by two spaces are one and the same item, and the first form met is the one that is kept, so the result shows “London”. That is why the example keeps “Sales@example.com” with its capital S. Untick “Ignore case” and the two spellings become different lines: the example would then return four addresses and remove one duplicate. Accents always matter: “café” and “cafe” stay separate.

Alphabetical or numeric sort

A plain alphabetical sort reads digits as characters and gives 10, 100, 2, because “1” comes before “2”. Here “A → Z” follows the rules of the page language and compares whole numbers by value, so you get 2, 10, 100, and “item 2” comes before “item 10”. “Numeric order” is meant for lists of numbers: it understands minus signs and decimals. The values 3, -10 and -5 come out as -10, -5, 3, and 1.5 and 1.25 come out as 1.25 then 1.5. Lines that are not numbers go to the end.

In a spreadsheet, you get the same result by selecting the column and running the remove duplicates command from the data menu.

Frequently asked questions

How long can the list be?

A list of 50,000 lines is handled without a noticeable wait on an ordinary computer. With very long lists, the result is recalculated after a short pause in typing.

How do I turn a column into a comma-separated list?

Choose “Comma” under “Result separator”. The items are written on one line with a comma and a space between them.

Can I sort a list without removing anything?

Yes. Untick “Remove duplicates” and every line is kept and simply sorted. The “duplicates removed” counter stays at 0, but “Show duplicates only” still tells you which items are repeated.

Why does “Shuffled at random” keep changing the order?

A new shuffle is drawn each time the list or an option changes.