Duplicate lines remover

Duplicate lines remover is a free tool that removes repeated lines from text and reports the original, remaining, and removed line counts.

Why deduplicate before importing a list?

Removing repeated lines before an import prevents the same complete entry from being processed more than once. This is useful when several lists have been combined or when copied data includes overlapping sections.

• Mail-merge recipients: Put one email address on each line before importing recipients into a mailing system. If each line contains several CSV fields, the entire row must match for it to count as a duplicate.
• Google Sheets exports: Clean a single-column list of product codes, order references, or email addresses after exporting or copying it from a sheet.
• GitHub configuration: Remove repeated package names, repository identifiers, or allowlist entries before placing them in a line-based configuration file.
• Form and CRM fields: Prepare values for fields that accept one category, tag, or identifier per line. Before pasting, confirm that the field accepts deduplicated input with one complete item per line.

This is line-based cleanup. It does not understand that two differently written records may refer to the same person, company, or product.

A short cleanup workflow

1. Place one item on each line and paste the text into the tool.
2. Run the duplicate removal, then review the cleaned text and the three reported counts.
3. Copy the result into the destination and confirm that the preserved line order and line-based output are appropriate.

The underlying operation uses PHP's array_unique function. Matching is based on complete line values, so capitalization, punctuation, and surrounding spaces can affect whether two entries match. A practical habit is to normalize the source first. For example, remove accidental trailing spaces and choose one capitalization style before deduplicating.

Example input:

SKU-104

SKU-220

SKU-104

SKU-318

SKU-220

Cleaned output:

SKU-104

SKU-220

SKU-318

Numbers are handled as part of each line. A value such as 104 is different from SKU-104, and 104. includes punctuation that changes the value.

What do the three result counters mean?

The counters show how much text entered the operation and how many line entries remain after duplicates are removed.

• Lines is the original line count.
• New lines is the line count after deduplication.
• Removed lines is the number of repeated entries taken out.

For the five-line SKU example, the cleaned list has three lines and two removed lines. The counters are useful for a quick check, but they do not verify that the remaining entries are valid for the system receiving them.

Which records should stay out of a line-based cleanup?

Keep structured or sensitive records out when whole-line comparison is not the operation you need. A CSV row with the same email address but a different name or status is a different line, so this tool will not merge the records or choose which field should win.

The same caution applies to program code, multiline addresses, and paragraphs. Repeated lines may be intentional, and removing them can change meaning or behavior. For field-aware cleanup, use a spreadsheet or database operation that targets a specific column.

The work is done on the server. Your input travels to the server over HTTPS and is not stored.

Frequently asked questions

Does the first or last duplicate remain?

PHP's array_unique keeps the first occurrence of a duplicate value and preserves its existing order. If sequence matters, arrange the source list before running the cleanup.

Are accented and unaccented words considered the same?

No. Characters such as é and e are different, as are accented words and their unaccented spellings. Normalize those characters separately if the destination treats them as equivalent.

Can this remove every blank line?

No. Deduplication can reduce repeated empty entries to one empty entry, but it is not a dedicated blank-line remover. Inspect the result and delete any remaining blank line if the destination does not permit one.

Should I use this or a spreadsheet formula?

Use this tool for a quick list where each complete line is one item. Use a spreadsheet function such as UNIQUE when you need to select a particular column, retain neighboring fields, or continue filtering and sorting the data.

How can I split combined text before removing repeats?

If the values are separated by commas, semicolons, or another delimiter instead of line breaks, convert that separator into line breaks first. The Text separator tool may help prepare the list, after which each resulting line can be compared as a separate item.