Xxhash Python

xxhash Python

Hashing is a protocol. Fix the algorithm, input byte encoding/framing, seed, digest width, and output representation before writing code. xxHash is fast and non-cryptographic; collisions and adversarial forgery remain possible.

Boundary

Use this skill when the project imports xxhash or explicitly requests xxHash for checksums, cache keys, deduplication candidates, sampling, or partitioning. Do not use it for credentials, password storage, tokens, digital signatures, HMAC, tamper-proof artifacts, untrusted integrity decisions, or security boundaries. Use a cryptographic hash/MAC/password KDF as the actual requirement dictates.

Know the contract objects

Read the hash contract before hashing structured values, text, files, or identifiers shared across systems.

Ordered workflow

1. Classify the requirement as non-adversarial checksum, lookup key, deduplication candidate, bucketing, sampling, or security. Stop and leave xxHash when authenticity or collision resistance matters.
2. Recover any existing protocol from stored digests, other languages, schemas, fixtures, algorithm name, width, seed, byte order, and text encoding. Never upgrade an algorithm silently.
3. Define a one-to-one byte serialization for the logical input. Encode text explicitly; frame multiple fields with lengths or a canonical serializer.
4. Choose width from collision consequences and population size. A wider digest lowers accidental collision probability but never makes xxHash cryptographic.
5. Use one-shot hashing for a materialized bytes object; stream chunks for large files/streams. Chunk boundaries must not affect the digest.
6. Choose bytes, hex, or integer output at the boundary and keep it consistent.
7. Test an authoritative vector, chunk invariance, seed/encoding sensitivity, and collision handling at the application layer.

Decision table

Canonical anchors

import xxhash


def fingerprint_text(value: str) -> str:
    payload = value.encode("utf-8")
    return xxhash.xxh3_128(payload, seed=0).hexdigest()

from pathlib import Path

import xxhash


def fingerprint_file(path: Path) -> str:
    digest = xxhash.xxh3_128(seed=0)
    with path.open("rb") as source:
        for chunk in iter(lambda: source.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()

Verify these constructors in the installed python-xxhash version before use. The same input bytes, algorithm, and seed produce the same digest whether updated once or in chunks.

Structured input framing

Naive concatenation is ambiguous: ("ab", "c") and ("a", "bc") both become b"abc". Encode field type/order and lengths, or use an already-specified canonical serialization:

def frame(parts: list[bytes]) -> bytes:
    return b"".join(len(part).to_bytes(8, "big") + part for part in parts)

Do not sort fields unless order is semantically irrelevant and the protocol states that normalization. Avoid platform-dependent repr, native integer byte order, locale encodings, and nondeterministic mapping/set iteration.

Collision and concurrency rules

• A digest match is a candidate match, not proof of equality. Compare original keys/content when false equality is consequential.
• Width selection is a probability/risk decision. Never promise “collision free,” including for 128 bits.
• Seeds are public deterministic parameters. They can separate namespaces but do not make the hash keyed or secure.
• Do not share one streaming object across threads to combine unordered chunks. Even where internal state is protected, update order determines the digest.
• digest() returns bytes, hexdigest() text, and intdigest() an integer. Do not infer byte order by casting; follow the package's documented representation.

Read streaming and testing and the security boundary.

Version grounding and completion

Inspect xxhash.VERSION, xxhash.XXHASH_VERSION, and available constructors in the project environment. The package was not installed in this foundry during authoring, so runtime examples are official-source-grounded but unexecuted here. Completion requires a named algorithm/width/seed, deterministic byte framing, fixed output representation, chunk-invariant tests, a collision resolution path where identity matters, and an explicit non-cryptographic boundary.

References

• Hash contract and framing
• Streaming and testing
• Security boundary