Encoding and character sets
Why do identical-looking strings compare differently?
Tools to use
Identical-looking text can contain different code-point sequences. When search or duplicate detection disagrees with your eyes, inspecting the contents can be faster than repeatedly retyping. Combine Unicode Inspector and Normalizer to identify the actual difference.
Do not merge everything that looks similar. Names, product labels, formulas, and identifiers may require distinctions. Normalization is neither spelling correction nor a universal unsafe-character filter. Use it to inform a deliberate equivalence policy.
Build é in two different ways
é
éThe first line is U+00E9; the second is U+0065 followed by U+0301. They normally look like é but contain one versus two code points. Inspect separately so the newline does not become part of the example. Changing fonts changes appearance, not the underlying sequence.
Normalize the second string: NFC composes it into é, while NFD keeps the decomposed e-plus-accent form. Both establish defined forms for comparison. Applying NFC again to an NFC result should not change it. Compare code points and UTF-8 byte counts before and after to explain the transformation.
Full-width Ａ and circled ① need another decision
Ａ①NFC leaves this input unchanged; NFKC turns it into A1. Compatibility normalization can help search matching, but it loses the circled styling and full-width distinction. You may need a normalized search key while retaining the original text for display.
NFKD also applies compatibility changes but uses decomposed forms. Do not choose whichever of the four outputs is shortest. The relevant systems must agree on a form and tolerate the distinctions it removes. Changing that policy later can affect existing IDs and search behavior.
Normalization does not necessarily remove invisible characters
ab
abInspection reveals U+200B between a and b. It occupies no visible width but remains a code point. Neither NFC nor NFKC turns this example into ab. Normalization is not a cleanup that removes every invisible character; an allowed-character policy is a separate concern.
Inspector categories help identify whitespace, combining marks, and control/format characters; they are not a complete name database or safety verdict. Zero-width characters can serve linguistic or rendering purposes. Do not equate invisible with malicious or disposable; test legitimate examples as well as the problematic one.
A lookalike a may belong to another script
a
аThe first is U+0061 and the second U+0430. They can look very similar, but belong to different scripts, and NFKC does not merge them. Normalized text is not automatically free of lookalikes used for impersonation. URL and account-name safety require separate checks.
Conversely, replacing every lookalike through an ad hoc table can corrupt legitimate names in other languages. Loose search matching and identity checks for login or authorization need not use the same equivalence rule. Inspect the characters here, then choose a policy appropriate to the service.
Changed positions are not an editing history
Changed positions compare code-point arrays by index. Decomposing one character shifts later positions and can mark several differences. These are not counts of editing actions or mistyped characters. Read the complete before/after sequences and their surrounding text.
UTF-16 positions and code-point positions can also differ, especially after emoji. JavaScript string offsets and code-point counts are different measures. When reconciling an error position, align the unit and whether counting starts at zero or one.
Identify the cause before choosing a transformation
Preserve the original, identify the differing code points, then determine whether NFC aligns them, compatibility conversion is needed, or normalization does not affect the difference. Decide which inputs may be transformed and which must remain original. Do not default to NFKC as a universal repair.
Record code-point sequences as well as visible examples. Keep composed/decomposed é, full-width Ａ, a zero-width character, and cross-script a as tests for both intended equivalence and required distinction. Reliable matching means fixing one failure without accidentally merging inputs that must remain separate.
Search equivalence is different from rewriting stored data
A product search may normalize comparison text so full-width Ａ finds ASCII A, without rewriting the official displayed name. Keeping source spelling separate from a search representation can improve matching while preserving data. Compare NFC and NFKC to decide which differences your use case treats as equivalent, not to rank one as universally better because it changes more or fewer characters.
Adding normalization later to user IDs, signed strings or passwords can change compatibility with established equality rules. Visual similarity is not permission to rewrite stored data. Inspect code points, compare normalization forms, then check the target system's comparison contract. Finding a cause and authorizing a data migration are separate decisions.
Build separate small cases for composed/decomposed é, full-width/ASCII A and a zero-width insertion. Mixing causes makes it harder to tell which transformation helped. Inspect code points and UTF-8 sizes as well as appearance. Defining a narrow, justified equivalence policy is safer than trying to make every visually similar string identical.