How to Remove Duplicate Lines from Text (Online, Excel, and the Command Line)

Duplicate lines creep into email lists, keyword exports, logs, and URL collections. Removing them by hand doesn't scale. A few settings decide whether the result is what you expect, so it helps to know them.

Decisions to make

• Keep first or last occurrence: most tools keep the first and drop later repeats.
• Preserve order or sort: deduplicating while keeping the original order is different from sorting the list and removing neighbors.
• Case sensitivity: is Apple the same as apple? For emails and URLs, you usually want case-insensitive matching (with care: URL paths can be case-sensitive).
• Whitespace: a trailing space makes two otherwise identical lines different. Trim before comparing.
• Empty lines: decide whether to remove blank lines too.

In a browser tool

1. 1Paste your list, one item per line.
2. 2Choose case sensitivity and whether to trim whitespace.
3. 3Remove duplicates and check the count of lines removed.
4. 4Copy the result. Do this locally if the list contains personal data such as email addresses.

In the terminal

# Keep first occurrences, preserve order
awk '!seen[$0]++' input.txt > output.txt

# Sort and remove duplicates
sort -u input.txt > output.txt

# Case-insensitive unique after sorting
sort -f input.txt | uniq -i

Note that uniq only removes adjacent duplicates, so the input must be sorted first, which is why sort -u is the common shortcut.

In code

// JavaScript: preserves first occurrence order
const unique = [...new Set(lines)];

# Python
unique = list(dict.fromkeys(lines))

In a spreadsheet

• Excel: Data, then Remove Duplicates, or use the UNIQUE function in current versions.
• Google Sheets: the UNIQUE function returns the distinct rows.