David Ondrej Risky Changes
A filter once passed every test but disabled ~99% of the feature on live data. Tests check code correctness; research and live measurement check whether a change is useful.
When this applies
Use for changes where being wrong is expensive or customer-visible:
- Public API fields, filters, or response shaping
- Dropping, transforming, or reordering upstream data
- Billing, pricing, caps, or quotas
- Defaults, thresholds, or provider request parameters
- Unverified assumptions about external data or user behavior
If unsure whether a change qualifies, apply this process.
1. Identify assumptions
List the assumptions the change depends on. Mark which have evidence and which are still unverified.
2. Research before implementing
Use available research tools and reliable sources. Investigate each distinct question separately, covering at least:
- How do leading products handle this design decision?
- What does real-world data look like: frequencies, shapes, and edge cases?
- What do users or agents actually need?
If evidence contradicts an assumption, reconsider the design before coding. If research is unavailable or inconclusive, state the gap; do not treat the assumption as verified. Do not ship while material assumptions remain unverified.
3. Measure real behavior
Run 10–20+ realistic cases against the real endpoint or provider:
- Base cases on real usage; vary topics, parameters, languages, and edge conditions.
- Define benchmarks per case: speed, quality, accuracy, and how often the new behavior occurs.
- Use hard numbers where possible. For subjective quality, use blind, criteria-based judging.
- Compare before and after when both can be measured.
- Read-only production analysis also counts as measurement.
Save the cases and results in the project's evals folder, e.g. docs/evals/YYYY-MM-DD-<endpoint>-<focus>.md. Create the folder if needed. Without this record, the change is not verified. Unit tests do not replace live measurement.
4. Confirm product-owner approval before shipping
Present decisions that affect what customers see or pay, with supporting research and measurements. Get the product owner's approval for those decisions; do not bury them in a plan or code default. Existing approval counts if it covers the actual behavior being shipped.
5. Verify after deployment
Within one day, measure the change on real traffic through read-only production analysis or a live sweep. Report results that differ from expectations immediately.