Amey-Thakur Binary data representation
Text hides representation decisions that binary exposes: byte order, alignment, signedness, and size. Cross-system binary data fails when two sides disagree about any of these, usually silently and with plausible wrong values.
Method
- Fix the byte order explicitly. Network protocols and file formats must specify endianness, and relying on the host's is what breaks when the data crosses machines.
- Specify exact widths. Integer sizes vary by platform, so fixed-width types are the only safe choice in a wire format.
- Be explicit about signedness. A negative number read as unsigned becomes enormous, which is a classic source of nonsensical values and allocation failures.
- Account for alignment and padding. Structures in memory contain padding that is not part of the logical data, so writing a structure directly to disk is not a portable format.
- Validate length fields before allocating. A corrupt or hostile length is the most common parsing vulnerability, and bounds checking comes before trust (see input-validation).
- Include a magic number and a version. Identifying the format and its version at the start makes every future change survivable.
- Read hex dumps when debugging. Looking at the actual bytes resolves in seconds what reasoning about the parser cannot.
Boundaries
Binary formats are compact and fast and cost readability and debuggability, which is why text formats persist for configuration. Established serialisation libraries handle these concerns and should be preferred to hand-rolled formats. Text encoding is a separate concern with its own pitfalls (see character-encoding).