Why identical-looking text produces different hashes

Worked example: invisible difference

Use this exact sample to isolate the behavior discussed above. Keep the operation and options with the sample when reporting a result; changing the input representation can change what the test demonstrates.

Input or question

A: hello
B: hello followed by LF

Expected result or interpretation

The SHA-256 digests differ because B contains an additional byte.

A non-breaking space can hide in copied text

Compare "A B" with "A B", where the second sample uses a non-breaking space. Both display a gap, but the UTF-8 representations differ and therefore their digests differ. Use an escaped code-point preview such as U+0020 versus U+00A0 to make the distinction explicit. If the application considers them equivalent for search, define a normalization policy before hashing normalized search keys. If it signs or identifies exact content, preserve the distinction. Do not replace every unusual space in a copied source simply to match an expected checksum. First identify which system introduced the character and whether the reference was calculated from the same artifact.

Inspect line endings and final terminators

LF is one byte in UTF-8; CRLF is two. An editor may also add a final newline when saving. State whether that terminator belongs to the canonical fixture. A browser textarea can normalize line endings, so this text generator should not be used to verify original file bytes. Use a local byte-aware tool for that job and preserve the source file during investigation.

Make invisible spaces observable

Ordinary spaces, tabs and non-breaking spaces have distinct representations. Trailing spaces may be invisible in a rendered page but remain in copied input. Compare escaped strings, character codes or byte arrays. Length is a useful clue, not proof: equal-length sequences can still differ. Do not globally trim a signed or byte-addressed payload merely because the output is inconvenient to compare.

Check Unicode composition and encoding

A precomposed accented character and a base character plus a combining mark may look alike but produce different UTF-8 bytes. NFC normalization can align canonical equivalents when explicitly required. Separately, UTF-16 files and UTF-8 files with a BOM contain different storage bytes from this tool’s UTF-8 output. Copying text does not preserve those file-level distinctions.

Define exact or canonical equality

Exact-byte workflows treat every changed byte as a mismatch. Normalized-text workflows need a versioned definition of transformations before hashing. Apply that definition identically to reference and candidate inputs. Changing it changes the identity scheme. Record which layer owns normalization so a later edit does not silently make old checksums incompatible. Use a line diff only after understanding any normalization it performs.

Inspect before canonicalizing

Save the mismatching inputs before applying any transformation. Create an escaped preview that reveals carriage returns, line feeds, tabs and non-breaking spaces. For a file, inspect its encoding and any byte-order mark with a byte-aware tool. Compare a small region around the suspected difference. This preserves evidence about which layer introduced the change instead of erasing it with trimming or newline replacement.

If the product intentionally compares normalized text, define a canonicalization version and test both reference and candidate through exactly the same process. Specify the order of operations, because trimming, Unicode normalization and line-ending conversion are distinct choices. Keep a fixture where a meaningful difference must remain meaningful after canonicalization. For example, removing all whitespace would make different words or token boundaries collapse together and would not be an acceptable general text policy. Record how older stored digests are handled if the rules change. In exact-byte workflows, do none of those transformations before comparison. The browser generator is useful for controlled text samples, but its textarea boundary means it should not be the final authority for an original file’s byte representation. Distinguish the artifact you can observe from the artifact named by the reference checksum.

Verification cases

• Ordinary vs non-breaking space: Different UTF-8 bytes despite similar appearance.
• Precomposed vs combining accent: May differ unless a normalization policy applies.
• UTF-8 with BOM vs no BOM: Original file hashes differ.

Decision reference

Before you use the result

1. Inspect line endings before changing algorithms.
2. Check trailing and non-breaking spaces.
3. Document canonicalization rules.
4. Keep browser text hashes separate from file hashes.

Continue the workflow

Create and compare SHA-256 checksums for text fixtures addresses verify exact text integrity with sha-256. Unicode regex matches and JavaScript UTF-16 indices addresses interpret regex offsets in unicode text.