ToolPetal
Duplicate line checker
Client-side only. Dedup runs in this tab. Nothing is uploaded.
Clean a messy list in one pass
Paste a column of emails, product codes, hashtags, or log hosts. Hit Remove duplicates. The output keeps one copy of each uniqueness key and preserves the order of the occurrences you chose to keep (first or last). Copy the result into your CRM import, SQL seed, or config file.
Typical jobs: mailing-list hygiene before a campaign, collapsing repeated SKUs from two exports, turning a noisy tag dump into a clean allowlist, or stripping repeated lines from a pasted log slice.
How this page decides "the same"
Each input row becomes a compare key. The options below change only the key. The text written to the output is always the original line text for the kept occurrence (not the lowercased or trimmed key).
| Option | Default | Effect on the compare key |
|---|---|---|
| Case sensitive | On | On: Alex and alex differ. Off: both fold to the same key. |
| Trim each line | On | On: leading and trailing spaces are ignored for matching. Off: spaces count. |
| Ignore empty lines | On | On: blank rows are dropped before counting. Off: blank rows can appear once in the output. |
| Keep first / last | First | First keeps the earliest original spelling. Last keeps the newest spelling for that key. |
Worked example
Leave the sample paste and the default options (case on, trim on, ignore empty, keep first). You should see:
- 10 original lines (including the blank and the padded
SKU-1002) - 5 unique lines after dedup
- 4 duplicates removed (the blank is ignored, not counted as a duplicate)
Expected unique output in order:
[email protected] SKU-1001 [email protected] SKU-1002 SKU-1003
Why: the blank is ignored. SKU-1002 matches SKU-1002 after trim, so the first spelling wins. The two trailing SKU-1001 rows and the second [email protected] are duplicates. Turn case sensitive off and re-run: [email protected] collapses into the first email, so unique count drops to 4.
When you should keep duplicates
- Audit trails: repeated log lines can mean a retry storm. Deduping hides the signal.
- Ordered events: the same user ID twice may be two real actions. Keep both unless you only need a set of IDs.
- Fuzzy near-matches:
Acme IncvsAcme Inc.stay different here. If you need fuzzy merge, use a spreadsheet or dedicated matching tool. - Whitespace-significant data: turn trim off when leading spaces are part of the value (rare, but real in some fixed-width exports).
Limits
- Exact line match after optional trim and case-fold only. No Levenshtein, no column-aware CSV parse, no regex.
- Very large pastes are limited by your browser tab memory. For multi-million-row files, use a local script or database DISTINCT.
- Refresh clears the boxes. Copy the result out if you need it later.
- Pair with word counter for length checks, case converter or camel case converter when you need a uniform casing pass, and slug generator when the cleaned list becomes URL paths.
FAQ
Does this upload my list?
No. Deduplication runs in JavaScript in this tab. ToolPetal has no server step for the duplicate line checker.
What does case sensitive mean here?
When case sensitive is on, [email protected] and [email protected] are different lines. When it is off, they share one uniqueness key (lowercased for compare only). The kept line still shows the original spelling of the occurrence you chose to keep.
Should I keep the first or last occurrence?
Keep first when the earliest row is the source of truth (most mailing-list and SKU cleanups). Keep last when a later paste is a corrected value and you want the newest spelling to win while still collapsing repeats.
Is this a fuzzy or regex matcher?
No. Lines match only after optional trim and optional case-folding. It does not ignore typos, reorder words, or run regex. For pattern work use a dedicated regex tester.