Amey-Thakur Character encoding
Encoding bugs look like corruption but are almost always a mismatch: one layer wrote in one encoding and another read in a different one. Getting this right is mostly about being explicit at every boundary and using UTF-8 everywhere you can.
Method
- Use UTF-8 throughout and declare it everywhere. Database, tables, columns, connection, HTTP headers, and file reads. A single layer defaulting to something else produces the classic garbled output.
- Know that a character is not a byte or a code unit. Length limits measured in bytes cut multi-byte characters in half, and a validator counting units mis-measures anything outside the basic range.
- Count what the user perceives when it matters. Emoji, combining accents, and flags are several code points rendered as one character, so a character limit needs grapheme awareness to behave sanely.
- Normalise on input. Deciding one normalisation form at the boundary makes comparison, deduplication, and search consistent (see locale-aware-sorting).
- Check the database column type actually supports full Unicode. Some engines have a legacy utf8 that stops short of the full range and silently rejects or truncates emoji.
- Validate rather than strip. Removing characters you did not expect mangles legitimate names; rejecting with a clear message is more honest than silent mutilation.
Boundaries
- Encoding correctness does not imply rendering correctness; a font still needs the glyphs (see text-expansion-layout).
- Legacy systems may genuinely require another encoding, in which case convert explicitly at the boundary and document it.
- Normalisation can alter strings that some systems treat as distinct, so apply it consistently rather than selectively.