Remove duplicate lines
Paste a list and get it back with repeats removed and the original order intact. You can match ignoring letter case, ignoring surrounding spaces or collapsing runs of whitespace, and choose whether the first or the last copy survives. A summary shows how many lines went and which ones repeated most.
How to use the Remove Duplicate Lines
- Paste your list, one item per line.
- Choose how strictly lines should be compared.
- Decide whether the first or last copy is kept — the position in the output stays the same either way.
- Copy the deduplicated list, or download it.
Order is preserved
The obvious way to remove duplicates is to sort and then discard neighbours that match, which is what the classic sort -u pipeline does — and it destroys the original order. That matters for a ranked keyword list, a log excerpt, a playlist or a set of steps. This tool keeps the sequence exactly as you pasted it and removes only the later repeats, so nothing is reshuffled. If you do want sorted output, run the result through the sort-lines tool afterwards.
What counts as a duplicate
Comparison is exact by default apart from leading and trailing spaces, which are trimmed because they are almost never meaningful and are the most common cause of "identical" lines that are not. Three options loosen it further:
| Option | These become duplicates |
|---|---|
| Ignore case | London and LONDON |
| Collapse whitespace | John Smith and John Smith |
| Trim (default on) | item and item |
Punctuation and accents are always significant: cafe and café are different lines, as are item, and item. Normalise those first if you need them merged.
First or last
When every duplicate is identical the choice makes no difference. It matters when lines differ only in the part you are ignoring — with case-insensitive matching on, keeping the first preserves iPhone while keeping the last would leave IPHONE. Keeping the last is also the right choice for a log where each repeat carries a newer timestamp. Either way the surviving line stays in the position where the first copy appeared.
Typical uses
Cleaning an email or phone list before an import, deduplicating a keyword export from a search tool where the same term arrives from several reports, tidying a bibliography, removing repeated log lines before reading them, and merging two lists by pasting one after the other and deduplicating the join. Because it runs in your browser, customer lists and internal logs never leave the machine.
Reading the summary
Underneath the output you get the input and output counts, how many lines were removed as a percentage, and the eight most-repeated lines with their counts. That table is often the interesting part: a value repeating hundreds of times usually means a broken export or a join that fanned out, not a data-entry slip.
Frequently asked questions
Does it sort the list?
No. The original order is preserved and only later repeats are removed. Use the sort-lines tool if you want sorted output as well.
Are "Apple" and "apple" the same line?
Only if you tick "ignore letter case". By default the comparison is case-sensitive.
What happens to blank lines?
They are treated as ordinary lines and deduplicated unless you tick "drop blank lines", which removes all of them.
How large a list can it handle?
Hundreds of thousands of lines are fine — matching uses a hash set, so the work grows linearly with the list.
Privacy
This tool runs entirely inside your browser using WebAssembly and the Canvas/File APIs. Your files are never uploaded to ToolFlint or any third party; you can verify this in your browser's network tab or by switching to airplane mode after the page loads. Read how we process files.
Last updated 2026-09-23.