Text and regular expressions
Remove duplicate lines, blank lines, and extra spaces
Tools to use
Names copied from email, extracted from logs, or collected manually often contain stray spaces and duplicates. Line Tools can prepare such lists for the next step. But a shorter, tidier list does not necessarily preserve every piece of information you need.
Use a fictional product-ID list to try trimming, empty-line removal, deduplication, and sorting separately. You do not need personal data or a real permissions list to understand the behavior. Start by preserving order and occurrences if either matters to your task.
Start without sorting the original list
item-10
item-2
item-2
item-10
item-1Initially disable trimming, empty-line removal, deduplication, and sorting, and leave numbering off. This establishes a baseline. Applying every cleanup at once makes it harder to explain which option removed which lines. Use the six-line sample and enable one operation at a time.
Trimming can create duplicates
Trimming alone makes lines 2 and 4 identical item-2 strings, and similarly aligns item-10, but both occurrences still remain. Next enable empty-line removal and deduplication while keeping sorting off: the result is item-10, item-2, item-1, in first-appearance order.
item-10
item-2
item-1Trimming happens before deduplication. Disable it if outer spaces distinguish records. Fixed-width data and indented prose can use whitespace as information. Once only the cleaned list remains, removed spaces and original occurrence positions cannot be reconstructed from that output.
Why item-10 can sort before item-2
Ordinary ascending text order gives item-1, item-10, item-2. Numeric sorting gives item-1, item-2, item-10. This can better match how people read numbered filenames, but it does not convert the IDs into numbers; only the comparison rule changes.
Sorting uses the runtime's Intl.Collator. Letter case, accents, and language-specific ordering do not necessarily follow byte order. Do not define a cross-system machine contract solely by this screen's ordering. If the original order expresses priority, keep sorting disabled.
Case-sensitive sorting is not duplicate matching
Alpha
alpha
AlphaDeduplication keeps Alpha and alpha as distinct strings. Changing sorting's case setting does not make duplicate removal case-insensitive. Confusing the two can make correctly retained entries look like failed cleanup. A small example establishes which operation an option affects.
If your business rule ignores case, confirm that it applies to these records. Similar filenames or identifiers may refer to different targets. List-set comparison provides a case-insensitive membership option, but matching there does not establish identity in the original system.
Numbering comes last, not from original positions
Numbering labels the cleaned output 1, 2, 3 from the top; it does not preserve positions in the original six-line input. With sorting enabled, numbers follow the sorted order. This helps review a list but is not a source-file line reference.
The prefix becomes part of the output text. Another tool will receive 1. item-10, not just item-10. For machine processing of IDs, unnumbered output may be appropriate. Separate presentation for people from the values required by the next operation.
Removing duplicates discards occurrence counts
An error repeated a hundred times becomes one line after deduplication. That is useful for listing error types, but loses frequency information. Orders, votes, and access records can likewise use identical strings for distinct events. Decide whether you need kinds or occurrences before treating repetition as unnecessary.
A quoted CSV field can contain a newline that is not a record boundary. Do not treat such data directly as one item per text line. Parse its structure in CSV Editor and select the relevant column first. Line Tools operates on newline-delimited strings, not CSV records.
Before handing it off, explain what was removed
Check the output against its next use: must order survive, should case variants remain distinct, can spaces be removed, and are line numbers now part of the values? A tidy output alone hides what was lost, so keep the original for comparison.
A note such as 'trimmed outer spaces, removed empty lines and exact duplicates, preserved first-appearance order' makes the result interpretable. Manage necessary originals and processing conditions in your own workflow rather than assuming the tool keeps input history. Cleanup is about passing the right meaning onward, not merely shortening a list.
The order of cleanup operations matters
Consider alpha, a space-padded alpha, and alpha again. Deduplicating without trimming leaves two distinct strings; trimming first leaves one. Neither is universally correct. Trimming may suit an ID list with accidental padding, while a test fixture for literal strings may need both forms. Decide whether whitespace is data or input noise before removing it.
Sorting has the same tradeoff. Alphabetical order may help a directory but can destroy a procedure or priority list. Added line numbers describe the cleaned output, not guaranteed source positions. Keep the original separately if provenance matters, and compare changes before discarding it. This tool does not maintain an audit history of the source.
Instead of enabling every option at once, inspect one change at a time: empty lines, padding, then duplicates and order. If the count surprises you, the last operation narrows the cause. A description such as 'trimmed, deduplicated by exact match, then sorted' makes the result reproducible. What was treated as equal and discarded matters more than a tidy appearance.