How to Remove or Extract Duplicate Lines with Textslave's Duplicate Lines Tool

Updated

Open Textslave in your browser, paste your text, and go to the Duplicate lines section. Choose Remove to delete repeated lines or Extract to keep only the lines that appear more than once. Leave "But do not collapse them" unchecked for a clean, deduplicated list; check it when you want to see every repeated occurrence. Use Preview to confirm the result before Save.

What the Duplicate lines section does

The Duplicate lines section contains two separate operations, each with its own preview and save controls:

Operation What it does Typical use
Remove Deletes duplicate lines from the text Cleaning a list, deduplicating exported data, tidying notes
Extract Keeps the duplicate lines (pulls them out) Finding repeated entries, spotting copy-paste errors

Both operations share the same option set, so once you understand one, the other behaves the same way.

Step-by-step: removing duplicate lines

  1. Paste your text into the Textslave input area. Each line should be a separate entry, since the tool works line by line.
  2. Go to Duplicate lines → Remove.
  3. Decide on the "But do not collapse them" checkbox.
    • Unchecked (default): duplicates are collapsed — each unique line appears once in the output.
    • Checked: duplicates are not collapsed — repeated lines stay as separate occurrences, so the output preserves the original repetition rather than reducing it to one copy.
  4. Optionally set advanced options (see the next section) to limit which lines are treated as candidates.
  5. Click Preview to see the result without committing.
  6. Click Save when the preview looks right.

What "But do not collapse them" actually changes

This is the option most people misread. Think of it as controlling whether repeated lines are merged into a single copy:

  • A list like apple / apple / banana with collapsing on becomes apple / banana.
  • The same list with "do not collapse" checked keeps the repeated apple entries as they are, so the operation does not reduce them to one line.

If your goal is a clean unique list, leave it unchecked. If you need to preserve the original count of repeated lines — for example, when the repetition itself carries meaning — check it.

Step-by-step: extracting duplicate lines

  1. Paste your text.
  2. Go to Duplicate lines → Extract.
  3. Choose whether to "But do not collapse them" — the same logic applies: unchecked gives you the duplicated lines in collapsed form, checked preserves each repeated occurrence.
  4. Preview, then Save.

Extract is the faster route when you don't want to delete anything but need to see which lines repeat — useful for checking a mailing list, a product SKU list, or a set of IDs for accidental repeats.

Narrowing the operation with advanced options

Both Remove and Extract offer the same advanced filters, so you can target only part of your text instead of the whole document.

Line-matching conditions

Limit the operation to lines that are:

  • equal to a string
  • different from a string
  • containing a string
  • lacking a string

Position filters

Apply the match everywhere, at the start, or at the end of the line.

For example, if you only want to deduplicate lines that contain @example.com, set the condition to containing and enter that string. Lines without it are left untouched, which is handy when one file mixes several kinds of records.

Verifying the result before you save

Always use Preview first. Check three things:

  1. Count — does the number of lines match what you expected after removal or extraction?
  2. Order — are the remaining lines in the order you need?
  3. Edge cases — did lines that differ only by trailing spaces or capitalization get treated as duplicates? The tool compares text lines as written, so near-identical lines may not be caught. If that matters, clean whitespace first with the Whitespaces & Tabs tools, then run the duplicate operation.

Common snags

  • Nothing was removed. Your "duplicates" probably differ by whitespace or case. Run Whitespaces & Tabs → Replace multiple whitespaces first, then retry.
  • Too much was removed. Check whether "But do not collapse them" was left unchecked when you wanted every occurrence kept, or whether an advanced filter was broader than intended.
  • Only some lines were affected. An advanced condition (equal to / containing / at the start, etc.) is still active — clear it and preview again.
  • Blank lines remain. Duplicate handling doesn't target empty lines; use Remove Empty line for those, including the option to also catch lines made only of whitespace.

Once the preview matches your intent, click Save to apply the transformation to your text.