Unicode regex matches and JavaScript UTF-16 indices

Decision reference

Separate the counting models

ASCII characters use one UTF-16 code unit and one UTF-8 byte, so simple examples hide the problem. An emoji outside the basic multilingual plane uses a surrogate pair in UTF-16 and usually four UTF-8 bytes. A visible accented character may be composed from multiple code points. Never label a regex index as a byte offset unless you have converted the preceding text to the same byte encoding.

Use Unicode mode deliberately

The u flag changes how several pattern constructs treat surrogate pairs and enables Unicode property escapes in supporting engines. It does not change match.index to a code-point count. A dot under u can consume a complete supplementary code point where a non-Unicode dot may consume one code unit. Keep the flags in your fixture so another developer can reproduce the observed match.

Understand the displayed index

For the subject "👋IH-1042", the code begins at UTF-16 index 2. That is a valid index for JavaScript slice, even though a person may see only one symbol before the code. If a Python service expects a code-point offset or a Rust service expects a byte position, translate explicitly. Sending the number 2 without an indexing contract is ambiguous.

Do not assume one code point is one glyph

Family emoji, flags, skin-tone modifiers and combining marks can contain several code points per visible symbol. Code-point-aware matching therefore does not solve all user-facing selection problems. Intl.Segmenter can help with grapheme segmentation in supporting runtimes. Use it when the product requirement concerns visible characters, and test fallback behavior rather than presenting code-unit counts as universally human-readable.

Normalize only under a documented policy

Visually similar Unicode text can have different underlying sequences. NFC normalization may make certain canonical equivalents easier to compare, but it changes the input representation. Do not normalize signed payloads or exact-byte artifacts without agreement. For human-entered search fields, choose and document a normalization policy, then use fixtures with both precomposed and combining sequences to verify the intended behavior.

Translate offsets only at an explicit boundary

If a browser sends an index to another runtime, include the indexing unit in the contract. To obtain a UTF-8 byte offset for a JavaScript match, encode the substring preceding match.index and count its bytes. To highlight in the same JavaScript string, use the UTF-16 offset directly. Do not convert twice or apply an offset from normalized text to an unnormalized original. The index is meaningful only for the exact subject representation used to calculate it.

Build a fixture with an ASCII prefix, an emoji prefix and a combining sequence prefix. Check start and end slicing for every match. A visible highlight should select the intended text without splitting a surrogate pair. For caret movement and truncation, consider grapheme segmentation rather than regex offsets alone. A family emoji can be one visible item assembled from several code points, so a code-point count still may not match user expectations. Keep byte counts, string lengths and grapheme counts labeled separately in diagnostic output. This prevents an otherwise correct number from being used in the wrong domain. If you normalize for search, keep a mapping strategy or avoid claiming offsets back into the original text unless you can reconstruct that relationship reliably.

Verification cases

• 👋IH-1042: Start is UTF-16 index 2, UTF-8 byte offset 4.
• ASCII prefix A: One unit in both the common ASCII counts.
• Combining accent sequence: Visible character count can differ from code-point count.

Worked example: offset after an emoji

Use this exact sample to isolate the behavior discussed above. Keep the operation and options with the sample when reporting a result; changing the input representation can change what the test demonstrates.

Input or question

Pattern: IH-\d+
Flags: gu
Text: 👋IH-1042

Expected result or interpretation

Match: IH-1042
UTF-16 start index: 2
UTF-8 byte offset: 4

Offsets change when text is normalized

Place a combining accent sequence before IH-1042 and record the JavaScript match index. Normalize the subject to NFC and repeat the search. The visible prefix may look unchanged while its UTF-16 length changes, so the two offsets need not agree. An offset calculated on the normalized string must not be applied directly to the original string for highlighting. Either display the normalized representation consistently or retain a reliable mapping between representations. The same principle applies when a backend strips whitespace before searching. Offsets belong to a specific subject, not just to the visible words it contains. Include that subject version in diagnostic reports.

Before you use the result

1. Document the indexing model at API boundaries.
2. Test supplementary emoji before a match.
3. Use grapheme segmentation for visible-character UI.
4. Normalize only with an explicit policy.

Continue the workflow

JavaScript regex flags explained through observable examples addresses choose regex flags deliberately. Encode Unicode text as Base64 without breaking emoji addresses fix base64 encoding of non-ascii text.