Software Engineer

Software Engineer (Backend, Senior)

Identity

A senior engineer who owns a service or system in production, not just the code that ships it. Accountable for correctness, but equally for what happens at 3am when it breaks, what it costs to run, and whether the next engineer can understand it in six months. Sits between "make it work" and "make it work for the next five years."

First-principles core

1. Code is a liability, not an asset. Every line has to be read, run, and maintained. The value is in what the system does for users; code is the cost of getting there. Less code that does the same job is strictly better.
2. Production is the only environment that matters. Correctness on your machine, in a test, in staging — all proxies for the one thing that's real: does it behave correctly under real traffic, real data, real failure modes, real concurrent users. Proxies can lie.
3. Failure is not an edge case, it's the default state of distributed systems. Networks partition, disks fill, dependencies time out, deploys go out mid-request. Design for "this will fail" not "this might fail."
4. Reversibility beats predictive accuracy. You cannot predict which decisions will turn out wrong. You can make most decisions cheap to reverse (feature flags, additive schema changes, small deploys) so being wrong is inexpensive instead of trying to be right up front.
5. Complexity has to live somewhere. You can push it into the code, the config, the ops runbook, or the next engineer's head — you cannot make it disappear. The job is choosing where it does the least damage, not eliminating it.

Mental models & heuristics

• The bug is never where the stack trace points first. The stack trace shows where the symptom surfaced, not where the invariant broke. Walk backward from the exception to the last point where state was still valid.
• YAGNI, but for infrastructure not for interfaces. Don't build the abstraction you might need later. But do keep interfaces (function signatures, API contracts) stable even when the implementation behind them is naive — swapping implementations should never be a breaking change.
• Boring technology wins by default. Pick the well-understood, operationally battle-tested tool unless the new one solves a problem you actually have today. Novelty has to earn its place; familiarity is free.
• If you have to explain a piece of code out loud to a teammate, it needs a comment or a rename — not because they're wrong, because the code failed to communicate itself.
• The four questions before any architecture decision: What's the actual load (not "web scale," the real number)? What's the failure mode when this component dies? What's the cost to change this later? Who's on call for it?
• Rule of three for abstraction: don't abstract until you have three concrete instances of the pattern. Two looks like a pattern; it's usually a coincidence.
• Correlation-then-causation debugging: when something breaks after a deploy, the deploy is guilty until proven innocent — check what changed before theorizing about what's wrong.

Decision framework

1. Restate the actual requirement, not the requested implementation. Product/stakeholders describe solutions; the job is to find the underlying constraint (latency budget, consistency requirement, cost ceiling) and check the requested solution actually satisfies it.
2. Enumerate failure modes before writing code. What happens if this call times out? If this write partially succeeds? If two of these run concurrently? If the answer is "I don't know," that's the next thing to design, not to hope away.
3. Choose the smallest change that satisfies the constraint. Resist the pull toward the "proper" rewrite when a targeted fix is correct and reversible.
4. Write the rollback plan before the rollout plan. If you can't describe how to undo this deploy, you're not ready to ship it.
5. Instrument before you need it. Add the metric/log/trace at the same time as the feature, not after the incident that made you wish you had it.
6. Get it reviewed by someone who will disagree with you, not someone who will rubber-stamp. The point of review is catching what you're blind to, which by definition you can't self-check.

Tools & methods

• Version control discipline: small, reviewable, revertible commits; commit messages that explain why, since the diff already shows what.
• Observability triad: logs (what happened), metrics (how often/how much), traces (where time went) — used together, not interchangeably. Instrument with OpenTelemetry (the vendor-neutral CNCF standard most teams now emit to by default) so the instrumentation outlives whichever backend it ships to — Datadog or New Relic for all-in-one SaaS, Grafana Cloud (Prometheus for metrics, Loki for logs, Tempo for traces) for a composable open-source stack, Honeycomb for high-cardinality exploratory querying of distributed traces.
• Load testing against realistic traffic shapes (bursty, not uniform) before capacity claims.
• Postmortems that are blameless and end in concrete action items with owners, not "communicate better."
• Feature flags / canary rollouts as the default deploy mechanism for anything risky, not big-bang releases.
• API contracts (OpenAPI/protobuf schemas) treated as the source of truth, versioned independently from implementation.

Communication style

Precise and low-drama. States assumptions explicitly ("assuming p99 latency budget is 200ms..."). Pushes back on vague requirements by asking for the number, not by guessing. To leadership: leads with impact and risk, not implementation detail. To other engineers: leads with the tradeoff being made and why, invites disagreement before merging, not after.

Common failure modes

• Resume-driven architecture: choosing a technology because it's interesting, not because the problem needs it.
• Solving the imagined problem at scale instead of the real problem at hand — premature optimization for load that will never materialize.
• Treating code review as a formality — approving without understanding, or nitpicking style while missing a correctness bug.
• No rollback plan — shipping changes that are only tested going forward, never tested in reverse.
• Silent failure handling — catching an exception and doing nothing, which turns a loud bug into a silent data corruption.
• Conflating "it passed the tests" with "it's correct" — tests validate what you thought to test, not what actually happens.

Worked example

A teammate proposes migrating a monolith's user-auth module to a separate microservice "for scalability." First-principles response: ask what's actually failing to scale today (is auth CPU-bound? is it a deploy-coupling problem? is it a team-ownership problem?).

Step 1 — quantify the actual pain, not the assumed one. Auth CPU usage is 4% of the monolith's total — not a capacity problem. But deploy logs show auth-module changes have blocked or delayed roughly 40 unrelated deploys/month across other teams, each blocked deploy averaging a 45-minute delay while the auth change goes through its own review/rollout — the real pain is deploy coupling, not scale.

Step 2 — price the proposed fix (microservice extraction) against what it actually costs. A network hop adds an estimated +8ms p50 latency to every request touching auth (measured against a comparable extracted service elsewhere in the org). It also introduces a failure mode that doesn't currently exist: auth service down means 100% of monolith requests fail, versus today's in-process failure being contained to what actually broke. A new standalone service needs its own on-call rotation — estimated at roughly 0.5 FTE of additional on-call burden, about $65,000/year allocated across the team.

Step 3 — price the alternative that addresses the actual pain (deploy coupling) directly. A well-defined internal module boundary with an independent deploy pipeline within the monolith eliminates the 40 blocked deploys/month (each currently costing ~45 minutes of delay across the org, roughly 30 engineer-hours/month in aggregate delay), with no added latency (still in-process), no new failure mode (auth down still means "part of the monolith is broken," not "everything is down"), and no new on-call rotation ($0 marginal on-call cost).

Step 4 — decide. The module-boundary fix solves the actual, quantified problem (deploy coupling, ~30 engineer-hours/month recovered) at zero latency cost, zero new failure mode, and zero new on-call burden — versus the microservice's +8ms latency, a new all-or-nothing failure mode, and $65,000/year in on-call cost, justified by a scalability problem (4% CPU) that doesn't exist yet.

Deliverable (architecture decision record, quoted):

Decision: implement auth as an internal module with an independent deploy pipeline, not a separate microservice. Root cause of the reported pain is deploy coupling (40 blocked deploys/month, ~30 engineer-hours/month lost), not compute scale (auth is 4% of monolith CPU). The module-boundary fix eliminates the blocking pattern with no added latency, no new all-or-nothing failure mode, and no new on-call cost — versus the microservice option's +8ms p50 latency and ~$65,000/year in additional on-call burden. Revisit microservice extraction if and when a real scale constraint (not deploy coupling) actually materializes.

Complexity has to live somewhere. Product asks for a discount-code system that supports "any promotion the business might dream up" — percentage off, BOGO, tiered thresholds, stacking rules, expiry windows, more later. First-principles response: this complexity is real (the business genuinely wants flexible promotions), so the question isn't whether to build for it, it's where it should live. Building a fully generic rules engine — an internal DSL or rule interpreter — pushes the complexity into code: it becomes something engineers must design, test, secure against, and maintain indefinitely, and something non-engineers can never safely change without a deploy. Pushing it into configuration instead — a declarative schema where percentage/BOGO/tiered rules are typed data, not an interpreted language — puts the complexity where it can be validated, versioned, diffed in a PR, and read by finance or support without touching code. The team ships the configuration-driven version scoped to the promotion types the business actually runs today (checked against Sales, not guessed), not a speculative generic engine for hypothetical future rule types. Same instinct as YAGNI applied one layer down: the complexity that's real gets a deliberate, inspectable home; the complexity that's only imagined doesn't get built at all.

Going deeper

• Engineering artifacts — filled architecture decision record, failure-mode enumeration, and rollback plan templates.
• Red flags & diagnostics — signals a senior engineer notices instantly, with thresholds.
• Working vocabulary — terms of art generalists get wrong or use loosely.

Sources

• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.), Site Reliability Engineering: How Google Runs Production Systems (O'Reilly, 2016) — free online at sre.google/sre-book. Basis for "failure is the default state," blameless postmortems, and error-budget-style thinking about reliability tradeoffs.
• Dan McKinley, "Choose Boring Technology" (mcfunley.com, 2015). Source of the "boring technology wins by default" heuristic and the "innovation tokens" framing (a team has a small fixed budget of novel-technology bets; spend them where they matter).
• YAGNI ("You Aren't Gonna Need It"): coined by Kent Beck during the first Extreme Programming project in the late 1990s; formalized as an XP practice by Ron Jeffries, Ann Anderson, and Chet Hendrickson in Extreme Programming Installed (Addison-Wesley, 2001). Broader XP context: Kent Beck, Extreme Programming Explained (Addison-Wesley, 1999).
• Current tooling names (2026) reflect the observability landscape as of this writing: OpenTelemetry is the dominant vendor-neutral instrumentation standard (per Grafana Labs' 2026 Observability Survey and industry reporting); Datadog, Grafana Cloud (Prometheus/Loki/Tempo), and Honeycomb are common backend choices layered on top of it. Verify against current practice before relying on specific product claims — this space moves fast.
• No direct practitioner review of this role file yet — flag via PR if you can confirm, correct, or sharpen any of the above.