Differential test old against new
A rewrite is safe only if it produces the same observable output as the original on real inputs. Run both side by side on a corpus and diff; every divergence is either a bug or an intentional change to document.
Procedure
- Keep the old implementation importable, renamed:
legacy.calc_totalvsnew.calc_total. Do not delete it until the diff is empty. - Build a corpus from production-shaped data — 1000+ inputs including the weird ones (zero, negative, max, unicode):
sqlite3 prod.db "SELECT * FROM orders ORDER BY random() LIMIT 1000" > corpus/orders.tsv
- Run both and record each pair:
for row in corpus:
a, b = legacy.calc_total(row), new.calc_total(row)
if a != b:
divergences.append((row, a, b))
- Classify each divergence: bug in new (fix it), bug in old that new intentionally changes (document and add to an allow-list with the reason), or undefined behavior in both (canonicalize the input).
- Shrink to the smallest divergent input and treat it like a fuzz crash (see
shrink-a-fuzz-crash-to-a-minimal-repro), then add it to the corpus so it stays checked. - Wire the diff into CI as a test that fails on any new divergence not in the allow-list:
grep -v -f accept.txt divergences.txt | wc -l # must be 0
- Only after the diff is empty modulo the allow-list delete the legacy implementation and its imports.
Pitfalls
- Comparing float outputs with
==produces false divergences. Compare to a fixed precision or exact integer cents. - A corpus of only happy inputs gives false confidence. Seed it with boundary and malformed cases.
- Nondeterministic outputs (timestamps, dict/map order) must be canonicalized before diffing, or you chase ghosts.
- Deleting the old code before the diff is empty removes the reference you need to explain a later failure.
Verification
python tools/diff_impl.py --corpus corpus/orders.tsv | grep -v -f accept.txt | wc -l
Passes = prints 0. Report: "new.calc_total matches legacy on 1000 corpus rows, 3 allow-listed intentional changes documented, legacy removed."