Technical Decision Making

Technical Decision Making (Verbose)

Core Patterns

Reversibility Governs Rigour

The single most useful question before analysing anything is: what does it cost to undo this? Bezos framed it as one-way and two-way doors, and the framing holds up because it maps directly onto how much process a decision deserves.

Two-way doors — reversible in days to a couple of weeks, with contained blast radius:

• A logging library, an HTTP client, a test framework
• A caching layer added in front of an existing store
• CI provider, linting rules, formatter
• Anything behind a feature flag with a working kill switch
• An internal API you control both sides of

One-way doors — reversible only at high cost, or not at all:

• Primary datastore and its data model, once a year of writes has landed in it
• A public API contract with external consumers
• Programming language / runtime for a service that will grow
• A vendor holding customer data (exit cost is migration plus legal)
• Anything that alters data irreversibly (a destructive backfill)
• Organisational splits encoded in service boundaries

Two things make this harder than it looks. First, decisions can be made two-way with modest effort: an adapter interface turns a vendor choice into a two-way door; an anti-corruption layer does the same for a framework; versioning an API from day one makes contract change reversible. Investing in reversibility is usually cheaper than investing in certainty.

Second, doors close over time. Choosing Kafka on day one is fairly reversible. Choosing Kafka after 40 services publish to it is not. The right question is not just "is this reversible" but "how long until it isn't, and what should we learn before then".

The asymmetry matters: on a two-way door, deciding fast and being wrong costs less than deciding slowly and being right, because the correction is cheap and you learn from production rather than from speculation.

The Weighted Decision Matrix

A matrix does not make the decision. It makes the disagreement explicit, which is what unblocks a stuck team. Two engineers arguing about Kafka versus SQS are usually not disagreeing about Kafka and SQS; they are disagreeing about how much weight operational burden deserves relative to throughput headroom. The matrix moves the argument to that question, where it can actually be resolved.

Worked example. Choosing a background job system for a Python API: ~200 jobs/sec peak, jobs are 50–500 ms, existing Postgres 15 and Redis in the stack, 4 engineers, no dedicated ops.

Criteria and weights agreed before scoring:

Scoring 1–5, with the scale defined per criterion (5 on ops burden = no new component; 1 = a new stateful cluster we must operate):

Reading it honestly:

• Postgres queue wins (4.15) largely because it adds nothing to operate and the team already knows Postgres. SELECT ... FOR UPDATE SKIP LOCKED is a well-trodden pattern at this scale.
• SQS (3.85) is within 0.3 — that is inside the noise of subjective 1–5 scoring. Treat the two as tied and decide on something the matrix does not capture (here: we want zero new IAM surface, so Postgres).
• Kafka scores badly (2.55) and the matrix shows exactly why: it is a log, not a job queue, and the team would be learning a stateful distributed system to solve a problem two simpler options already solve.
• Throughput headroom at 0.10 is doing almost nothing. That is correct — every option clears 200/sec — but it is worth noticing, because a criterion that cannot discriminate is decoration.

Rules for keeping this honest:

1. Agree weights before you score. Weights chosen after scoring are how people launder a preference into an analysis.
2. Define each scale. "3 on observability" must mean something specific.
3. Include the incumbent / do-nothing option. It often wins, and omitting it biases everything.
4. Cap at 5–7 criteria. Beyond that, everything gets a middling score and the totals converge.
5. If your preferred option loses, either change your mind or say which weight you think is wrong. Do not quietly re-score.

Spike Before Deciding — But Only for the Load-Bearing Unknown

A spike is a time-boxed experiment to resolve a specific factual uncertainty. It is not "let's play with it for a week". Every spike needs three things: the question, the box, and the decision rule.

## Spike: Postgres queue throughput
Question: Can `SELECT ... FOR UPDATE SKIP LOCKED` on db.r6g.large sustain
  200 jobs/sec claim-rate alongside current OLTP write load?
Time-box: 2 days.
Method: replay a production write trace, add synthetic job load, measure
  claim latency p99 and primary CPU.
Decision rule:
  - p99 claim < 20 ms and primary CPU delta < 15%  -> Postgres queue
  - otherwise                                       -> SQS
Result (2026-02-11): p99 claim 7 ms at 200/s, 340/s before p99 exceeded 20 ms;
  CPU delta 9%. -> Postgres queue. Headroom ~1.7x, below our 5x growth case,
  so recorded as a revisit trigger.

Decide without spiking when the options are similar in cost, the choice is a two-way door, or the uncertainty is about preference rather than fact — no experiment resolves "which API do we find nicer". Spike when a single number would flip the decision and you do not have that number.

The failure mode is the spike as procrastination: no question, no box, no rule, and at the end a team that feels informed but still cannot state why one option won. If you cannot write the decision rule up front, the spike will not decide anything.

Writing Down What Would Change Your Mind

Record falsifiers at decision time. This is the highest-leverage habit in the whole practice and the most commonly skipped.

## Revisit triggers (ADR-0033, Postgres job queue)
- Sustained job rate > 500/sec for 7 consecutive days (measured: 1.7x headroom)
- Queue-wait p99 > 5s for 3 consecutive days
- Job table > 50M rows or vacuum falling behind
- A second service needs to consume the same job stream
Owner: platform. Reviewed at each quarterly architecture review.

Why up front and not later: after a system is built, everyone involved is motivated to conclude it is still the right choice. Sunk cost is not a bias you can talk yourself out of by being aware of it. A trigger written before you were invested is evidence from a less biased version of your team.

It also disarms the retrospective argument. "We should never have used Postgres for this" becomes a checkable claim: did a trigger fire? If yes, the decision was sound and we should have acted on the trigger. If no, the complaint is aesthetic.

Wire triggers to something that observes them — a dashboard panel, an alert, a recurring review — or they are just prose.

Disagree and Commit

Teams need a way to stop deliberating that is neither consensus (too slow, and it rewards whoever is most stubborn) nor authority alone (fast, but wastes the dissent). Disagree-and-commit is that mechanism, and it has preconditions people routinely skip.

It works when:

1. There is a named decider, known before the discussion, not discovered afterwards.
2. Dissent is heard first. Committing before objections are on the table is just compliance, and the objection resurfaces as passive resistance.
3. The objection is recorded, in the ADR, in the objector's words. This is the load-bearing part. It converts "I don't like this" into a specific prediction that can be checked.
4. Commitment is real. The dissenter implements the chosen option properly. No shadow prototype, no deliberate under-investment, no "well, I said so" in the incident review.
5. New evidence reopens it; repetition does not. Restating the same objection with no new information is not reopening a decision.

In the ADR:

## Dissent
J. Okonkwo argued for SQS on the grounds that a Postgres queue couples job
throughput to primary-database health, so a slow query in the API can stall
fulfilment. Recorded and accepted as a known risk; mitigated by a separate
connection pool capped at 20 and a queue-wait alert. Reconsider if the coupling
causes any customer-visible incident.

That paragraph is worth more than the agreement that surrounds it. If a slow query does stall fulfilment eighteen months later, the team does not argue about who saw it coming — the mitigation and the reversal condition are already written down.

Sizing the Effort

A rough calibration for how much process a decision earns:

If a decision is taking effort wildly out of proportion to its row here, that mismatch is the thing to fix first.

Common Anti-Patterns

❌ Treating every decision as irreversible. Three weeks and four meetings to choose a date library. ✅ Classify the door first. Two-way doors get decided by whoever is doing the work, in hours.

❌ Treating irreversible decisions as reversible. The data model that will carry two years of writes gets chosen in a PR at 6pm. ✅ One-way doors get a spike, a matrix, an ADR, and a second reader.

❌ The reverse-fitted matrix. Weights adjusted until the preferred option wins, then presented as analysis. ✅ Fix weights before scoring. If your option loses, name the weight you think is wrong and defend that instead.

❌ The unbounded spike. "Let's prototype it and see" with no question, box, or decision rule; two weeks later the team knows more and has decided nothing. ✅ Write the question and the decision rule before you write any code.

❌ No named decider. Discussion circles until the most persistent person wins, or until someone merges code that settles it by default. ✅ Name the decider up front. Ambiguous ownership does not avoid a decision, it just makes the decision arbitrary.

❌ Consensus as the bar. Everyone must agree, so the team optimises for the least objectionable option and takes weeks to reach it. ✅ Decider decides, dissent is recorded, everyone commits.

❌ Committing without recording dissent. The objection goes underground and returns as half-hearted implementation or a shadow project. ✅ Write the objection into the ADR, in the objector's words, with a reversal condition.

❌ No revisit triggers. The decision is either defended forever or abandoned in a fit of frustration, both without evidence. ✅ Write the falsifier at decision time and put it on a dashboard.

❌ Resume-driven selection. The technology is interesting rather than suitable; the matrix criteria quietly mirror its strengths. ✅ Derive criteria from the requirements before looking at any candidate.

❌ Ignoring the do-nothing option. "Should we adopt X" is asked without "what if we keep doing what we do", which is often the highest-scoring row. ✅ Always score the incumbent.

❌ Deciding with no exit cost estimate. A vendor is chosen without anyone asking what leaving would take. ✅ For any one-way door, write down the migration path out, even roughly.

Technical Decision Making Checklist

• Reversibility classified: one-way or two-way door
• Effort proportional to that classification
• A named decider, agreed before the discussion started
• The do-nothing / incumbent option is on the list
• Criteria derived from requirements, not from a favoured candidate
• Weights agreed and written down before any scoring
• Each score scale defined so scores mean something
• Options within ~0.3 treated as tied, not as a winner
• Spikes have a question, a time-box, and a decision rule
• Spike results recorded, including the ones that changed nothing
• Exit cost estimated for anything hard to reverse
• Decision recorded as an ADR with alternatives and consequences
• Revisit triggers written at decision time, with numbers
• Triggers wired to a dashboard, alert, or scheduled review
• Dissent heard before commitment and recorded verbatim
• Everyone, including dissenters, implementing the chosen option properly
• Decision communicated to the people who must live with it