Shrink Vector Store

Shrink Vector Store

Reduce a stored embedding index's footprint with scalar (int8) or binary quantization plus an asymmetric float rescore, so recall stays high while the store shrinks 4–32×. Vectors change; provenance metadata does not.

Prerequisites

• numpy (only hard dependency). Python 3.
• The original float32 vectors (or a representative sample ≥10k) and a query set — required to measure recall; this skill cannot validate blind.

Decision (commit to one)

Pick int8+rescore unless the user explicitly needs maximum shrink. Do not offer both and ask — choose, state which, proceed.

Workflow

1. Confirm scope. Verify it is a storage-size problem, not an ingestion failure (see above). If ingestion, stop and explain.
2. Locate the float baseline. You need the original float32 vectors (or a sample ≥10k) and a query set to measure recall against.
3. Quantize + build index. Run the reference script:

   python scripts/quantize_embeddings.py --vectors vecs.npy --queries q.npy \
       --precision int8 --rescore --out store/
   

   --precision {int8,binary}; --rescore keeps a small float/int8 copy for the second-pass reorder; --matryoshka N optionally truncates dims.
4. Validate recall (mandatory feedback loop). The script reports recall@10 of the quantized+rescore store vs the exact float baseline. Gate: if recall@10 drops more than the user's tolerance (default 2%), do not ship — fall back to a higher-precision setting and re-run.
5. Preserve provenance. Only the vector column is replaced. Any id / source / timestamp / provenance fields MUST be carried through unchanged. The script asserts row-count and id alignment; if it fails, stop — never silently drop provenance.
6. Report. State: precision used, achieved shrink ×, recall@10 vs float, and whether the recall gate passed.

Rules

• R0 — Storage problem only. This skill addresses store size. An embed-time failure (provider rejects input as "too many tokens") is an upstream ingestion-chunking bug — quantization changes nothing about it. If that is the situation, say so and stop.
• R1 — Asymmetric distance. The query stays full precision; only the stored DB vectors are quantized. Never quantize the query.
• R2 — Rescore is not optional for binary. Binary without a rescore pass loses too much recall; the script enforces --rescore when --precision binary.
• R3 — Provenance is immutable. Quantization replaces vectors only. Row count, ordering, and all non-vector columns are preserved and asserted. A mismatch is a hard stop.
• R4 — Measure, don't assume. Always report measured recall@10 vs the exact float baseline on the user's own data. The "~96–99%" figures are priors, not guarantees for a given corpus.

TurboQuant 4-bit path — DARK, gated (do not enable)

A 4-bit TurboQuant-style path (training-free random rotation + per-coordinate scalar quant) can beat int8 on the footprint/recall frontier. It is deliberately not implemented here. It is gated on an external decision (port vetting + a measured benchmark beating the int8-rescore baseline + an IP/patent review). Until that gate explicitly passes:

• Do not add a 4-bit path to the script.
• Do not vendor a TurboQuant port into this skill.
• If asked for it, explain it is gated and that int8-rescore is the sanctioned baseline it must first be measured against.

Output

A quantized store directory plus a stdout report. Concrete example (int8 + rescore, 2000×128 sample):

precision=int8 rescore=True matryoshka=off
vectors=2000 dim=128 float_MB=1.0 quant_MB=0.3 shrink=4.0x
recall@10=0.9800 (gate: >= 0.9800)
written to <out>/
recall gate PASSED

Exit codes: 0 pass · 2 bad args/precondition · 3 provenance misalignment · 4 recall gate failed. A non-zero exit means do not ship the quantized store.

Limitations

• Needs the original float vectors (or a representative sample) to measure recall — cannot validate blind.
• Matryoshka dim-trim only works if the embedding model was trained for it; the script errors clearly if truncation collapses recall.
• Cross-environment determinism: if any rotation/projection is used, build-time and query-time must use the same seeded transform.

Reference

• scripts/quantize_embeddings.py — run it (don't read it): quantizes, builds the index, runs the asymmetric rescore, and prints the recall@10 vs float gate result. Handles its own errors and exits non-zero on a failed provenance or recall assertion.