Unicode Inspector

A string that looks identical to another string but compares unequal almost always has something invisible in it, or is normalised differently. This lays out every character with its code point, its bytes and its type, and marks the ones you cannot see.

Unicode Inspector — Code Points, UTF-8 Bytes and Invisible CharactersBuildFigure

Four different counts, and which one you meant

Ask how long a string is and there are at least four defensible answers, which is why this tool reports all of them.

A grapheme cluster is what a person points at and calls a character. It is what a cursor moves over and what backspace deletes. Code points are the numbers Unicode assigns; one visible character can be built from several. UTF-16 code units are what JavaScript's .length counts, and what most database and API length limits historically counted. UTF-8 bytes are what the string costs in a file, in a request body, or in a column declared in bytes.

For plain English all four agree, which is exactly why the distinction stays invisible until it suddenly matters. "a" is one of everything. "👩‍💻" is one grapheme cluster, three code points, five UTF-16 units and eleven UTF-8 bytes.

Surrogate pairs

UTF-16 encodes code points up to U+FFFF in one 16-bit unit. Everything above that — emoji, musical notation, most historic scripts, a good number of rare Han characters — is encoded as two units from a reserved range, a high surrogate in D800–DBFF followed by a low surrogate in DC00–DFFF. That pair is not two characters; it is one character stored in two slots.

Inspect a single 💻 above and the UTF-16 column shows D83D DCBB for the one code point U+1F4BB. The consequences are the ones you have probably hit: "💻".length is 2, charAt(0) returns half a character that renders as a replacement box, a naive slice can cut a string between the two halves, and a reversal algorithm that walks the string by index produces garbage. Iterating with for...of, Array.from or the spread operator walks code points instead and avoids all of it — but note that this still splits grapheme clusters, so it fixes surrogate pairs without fixing emoji sequences.

Combining sequences

An accented letter has two legal spellings. Precomposed, é is the single code point U+00E9. Decomposed, it is U+0065 (e) followed by U+0301 (combining acute accent) — two code points that render as one glyph. They look identical, they are not equal under ===, and one of them will not be found by a search for the other.

This matters far beyond accents. macOS historically stored filenames decomposed while Windows and Linux stored them composed, so a file copied between them can appear twice in a listing or refuse to open. Text pasted out of a PDF is often decomposed. Korean text can be stored as precomposed syllables or as separate jamo, which is the same problem with a different alphabet.

The fix is normalisation, and the rule of thumb is short: normalise to NFC before you store, compare or index anything. NFC composes what can be composed and is what the web platform assumes. NFD goes the other way, which is occasionally useful — stripping accents for a search index is s.normalize('NFD').replace(/\p{M}/gu, ''), and that only works because NFD separates the marks out first.

ZWJ sequences, and why "clean" is not always safe

U+200D, the zero width joiner, tells the renderer to fuse adjacent glyphs into one. Modern emoji are built out of it: 👩‍💻 is 👩 + ZWJ + 💻, and a family emoji chains several people the same way. Flags use a different mechanism again — two regional indicator letters, so 🇰🇷 is the pair RI-K and RI-R, no joiner involved. Skin tone is a modifier code point appended to the base, and U+FE0F selects the colour emoji rendering of a character that also has a monochrome text form.

The same joiner is genuinely required in Arabic and in Indic scripts, where ZWJ and ZWNJ control ligature formation and removing them changes what the word looks like. So the cleaning mode here is a scalpel, not a broom: run Inspect first, look at what is actually flagged, and strip only the characters you can account for. If a string contains one zero-width space in the middle of an English word, delete it. If it contains joiners between pictographs, leave them alone.

The invisibles that actually cause tickets

No-break space, U+00A0, is the most common by a wide margin. It arrives from copy-pasted web pages and word processors, it looks exactly like a space, and it defeats trim(), spreadsheet TRIM, and any regex written as / /. The BOM, U+FEFF, prepends itself to files saved as "UTF-8 with BOM" and shows up as a stray character before the first key of a JSON file or the first column header of a CSV. Soft hyphen, U+00AD, hides inside words copied from justified text. And the bidirectional override characters, U+202A through U+202E, can reorder how a filename displays without changing what it is — the basis of both a filename spoofing trick and the Trojan Source attack on code review.

When a comparison fails against two strings that look the same on screen, paste both here. One of them will differ, and the table will say where.

Questions people ask

Why does one emoji show up as several rows in the table?

Because it is several code points. The table is a code point view by design, and the count panel above it reports grapheme clusters separately so you can see both numbers at once. A composed emoji is typically a base pictograph plus a joiner plus another pictograph, sometimes with a skin tone modifier or a U+FE0F variation selector attached. All of those are real, individually addressable code points that happen to render as one glyph.

I cleaned the string and the words ran together. What happened?

The exotic-space option was switched off, so no-break spaces and their relatives were deleted rather than converted. Those characters are invisible but they are still spaces, so removing them closes the gap. Leave the checkbox on unless you specifically want every one of them gone.

Which normalisation form should I store?

NFC, in nearly every case. It is what HTML, URLs and most web APIs assume, it is the shorter representation, and it means equality comparisons behave the way users expect. Normalise on input at the boundary of your system rather than at comparison time, so that what is in the database is already canonical. The compatibility forms NFKC and NFKD are a different tool entirely — they fold fi into fi and ① into 1, which is useful for search matching and destructive for storage.

Is the type column the official Unicode character name?

No. It is a classification derived from script and category properties — Latin, Han, combining mark, format character and so on. The one place real names appear is the invisible characters, where the actual Unicode name is shown because that is the piece of information you need in order to decide whether to delete it. For a full character name, look the code point up in the Unicode charts.

Related