Operating Production Services
Overview
Building it is half the job; keeping it alive under real traffic and over time is the other half. This skill covers two areas the build checklist only name-drops: runtime reliability controls and Day-2 operations/maintenance.
Core principle: Assume everything fails — slow networks, traffic spikes, bad deploys, abusive clients, aging dependencies. Design controls so failure degrades gracefully and recovery is routine, not heroic.
Announce when relevant: "Using operating-production-services for runtime controls and maintenance."
A. Runtime Reliability Controls
| Control |
What / how |
Bar to clear |
| Rate limiting |
Cap requests per client key (user/IP/API-key). Algorithm: token bucket or sliding window. Return 429 + Retry-After header. Apply at edge/gateway AND sensitive endpoints (login, signup, write APIs) |
Abusive client throttled, normal user unaffected; limits documented |
| Throttling / quotas |
Per-tenant quotas; backpressure when overloaded; load shedding (drop low-priority work) before total collapse |
System degrades, doesn't crash, under overload |
| Timeouts |
Every outbound call (DB, API, cache) has a timeout. No unbounded waits |
No request hangs forever |
| Retries |
Retry only idempotent ops; exponential backoff + jitter; cap attempts; never retry 4xx except 408/429 (honor Retry-After) |
No retry storms; transient errors recover |
| Circuit breaker |
Trip open after repeated downstream failures; fail fast; half-open probe to recover |
One sick dependency doesn't cascade |
| Idempotency |
Idempotency keys on writes/payments so retries don't double-charge/double-create |
Duplicate request = single effect |
| Caching |
Client/CDN/server/DB layers; explicit invalidation + TTL; stampede protection |
Cache never serves stale silently; no thundering herd |
| Graceful shutdown |
On deploy/restart: stop taking new work, drain in-flight, close connections cleanly |
Zero dropped requests on redeploy |
| Health/readiness |
Liveness (am I up) + readiness (can I serve) + startup probe for slow-init apps; load balancer respects them |
Unready instance gets no traffic; slow boot not killed |
| Autoscaling |
Scale on real signal (CPU/queue depth/latency); min/max bounds; scale-in cooldown/stabilization to prevent flapping |
Spikes absorbed; no thrash; no runaway cost |
| Edge protection |
WAF, DDoS mitigation, bot filtering at the perimeter |
Common attacks blocked before app |
B. Day-2 Operations & Maintenance
| Area |
What / cadence |
Bar to clear |
| Monitoring |
Metrics (latency, error rate, throughput, saturation — the "golden signals") + dashboards |
Can answer "is it healthy right now" in 10s |
| Alerting |
Alert on symptoms (SLO burn-rate breach), not every blip; every alert maps to a runbook; route to a person; no alert fatigue |
Real problems page someone; noise suppressed |
| Tracing |
Distributed tracing with propagated request IDs across services, not just metrics/logs |
Can localize cross-service latency/errors |
| SLO / error budget |
Define target (e.g. 99.9% / p95 latency); track budget; freeze risky changes when budget spent |
Reliability is measured, not vibes |
| On-call & runbook |
Who responds; written runbook per common failure (symptom → diagnosis → fix) |
New engineer can resolve a page from the runbook |
| Incident response |
Detect → mitigate → communicate (status page) → blameless postmortem with action items |
Every incident produces a prevention task |
| Deploys |
Progressive: canary or blue-green; automated rollback on error-rate spike; never big-bang prod |
Bad deploy auto-reverts; no manual ftp |
| Schema migrations |
Backward-compatible expand→migrate→contract; no locking DDL on hot tables; decoupled from app deploy; reversible |
Migration can't break running old code; no long table lock |
| Disaster recovery |
Define RTO/RPO; test region/AZ failover; backups ≠ DR; documented recovery procedure |
Failover drill meets RTO/RPO |
| Backups & restore |
Automated backups, encrypted at rest + access-controlled; periodically test restore (a backup never restored = no backup); offsite copy |
Restore drill passes on a schedule |
| Cert & secret rotation |
Automate TLS cert renewal; alert on cert/secret expiry N days out; rotate secrets on schedule |
No expired-cert outage; rotation tested |
| Queue / async health |
Dead-letter queue for poison messages past retry cap; alert on DLQ depth; monitor lag |
Stuck messages isolated, not silently lost |
| Database upkeep |
Index health, vacuum/analyze, slow-query review, connection-pool limits, archival of old rows |
Queries stay fast as data grows |
| Dependency patching |
Regular CVE scan + update cadence; pin + lockfile; test before bumping; track EOL/deprecated |
No unpatched critical CVE lingering |
| Log & data retention |
Log rotation + retention policy; scheduled cleanup jobs for expired/temp data; PII retention limits |
Disks don't fill; data isn't hoarded forever |
| Feature flags |
Decouple deploy from release; kill-switch for risky features; clean up stale flags |
Can disable a feature without redeploy |
| Cost monitoring |
Ongoing spend dashboards + budget alarms; catch runaway resources/queries |
No surprise bill; cost tracked like latency |
| Capacity & deprecation |
Periodic capacity review vs growth; planned sunset path for old APIs/versions with notice |
Scale ahead of the wall; clean retirement |
Pre-Launch Ops Gate
Do NOT call a live service production-ready until:
Common Mistakes
| Mistake |
Fix |
| Rate limit only on login, not write APIs |
Limit every abusable endpoint; return 429 + Retry-After |
| Retry without backoff/jitter |
Retry storm DOSes your own backend — add backoff + jitter, cap attempts |
| Retrying non-idempotent writes |
Add idempotency keys first, or don't retry |
| "We have backups" (never restored) |
A backup never test-restored is not a backup — drill it |
| Alert on everything |
Alert fatigue hides real fires — alert on SLO symptoms |
| Big-bang prod deploy |
Canary/blue-green + auto-rollback |
| Caching with no invalidation plan |
Define TTL + invalidation before adding cache |
Companion skills
- Overall gate →
disciplined-delivery (its "observable" + DoD reference this)
- Web build →
shipping-production-websites (pillars 9/10/13 = this skill's depth)
- Security perimeter overlaps →
securing-applications (rate-limit also a security control; WAF, A09 logging)
1---2name: operating-production-services3description: Use when a system will run live and must stay up — designing traffic controls (rate limiting, throttling, timeouts, retries, circuit breakers, idempotency, caching, graceful shutdown, autoscaling) and Day-2 operations/maintenance (monitoring, alerting, SLO, on-call, incident response, deploys/rollback, backups, DB upkeep, dependency patching, log/data retention, cost). Triggers before launch and for any "how do we keep it running" concern. Stack-agnostic.4---56# Operating Production Services78## Overview910Building it is half the job; **keeping it alive under real traffic and over time** is the other half. This skill covers two areas the build checklist only name-drops: **runtime reliability controls** and **Day-2 operations/maintenance**.1112**Core principle:** Assume everything fails — slow networks, traffic spikes, bad deploys, abusive clients, aging dependencies. Design controls so failure degrades gracefully and recovery is routine, not heroic.1314**Announce when relevant:** "Using operating-production-services for runtime controls and maintenance."1516## A. Runtime Reliability Controls1718| Control | What / how | Bar to clear |19|---|---|---|20| **Rate limiting** | Cap requests per client key (user/IP/API-key). Algorithm: token bucket or sliding window. Return **429 + `Retry-After`** header. Apply at edge/gateway AND sensitive endpoints (login, signup, write APIs) | Abusive client throttled, normal user unaffected; limits documented |21| **Throttling / quotas** | Per-tenant quotas; backpressure when overloaded; **load shedding** (drop low-priority work) before total collapse | System degrades, doesn't crash, under overload |22| **Timeouts** | Every outbound call (DB, API, cache) has a timeout. No unbounded waits | No request hangs forever |23| **Retries** | Retry only idempotent ops; **exponential backoff + jitter**; cap attempts; never retry 4xx **except 408/429** (honor `Retry-After`) | No retry storms; transient errors recover |24| **Circuit breaker** | Trip open after repeated downstream failures; fail fast; half-open probe to recover | One sick dependency doesn't cascade |25| **Idempotency** | Idempotency keys on writes/payments so retries don't double-charge/double-create | Duplicate request = single effect |26| **Caching** | Client/CDN/server/DB layers; **explicit invalidation + TTL**; stampede protection | Cache never serves stale silently; no thundering herd |27| **Graceful shutdown** | On deploy/restart: stop taking new work, drain in-flight, close connections cleanly | Zero dropped requests on redeploy |28| **Health/readiness** | Liveness (am I up) + readiness (can I serve) + **startup probe** for slow-init apps; load balancer respects them | Unready instance gets no traffic; slow boot not killed |29| **Autoscaling** | Scale on real signal (CPU/queue depth/latency); min/max bounds; **scale-in cooldown/stabilization** to prevent flapping | Spikes absorbed; no thrash; no runaway cost |30| **Edge protection** | WAF, DDoS mitigation, bot filtering at the perimeter | Common attacks blocked before app |3132## B. Day-2 Operations & Maintenance3334| Area | What / cadence | Bar to clear |35|---|---|---|36| **Monitoring** | Metrics (latency, error rate, throughput, saturation — the "golden signals") + dashboards | Can answer "is it healthy right now" in 10s |37| **Alerting** | Alert on symptoms (SLO burn-rate breach), not every blip; **every alert maps to a runbook**; route to a person; no alert fatigue | Real problems page someone; noise suppressed |38| **Tracing** | Distributed tracing with propagated request IDs across services, not just metrics/logs | Can localize cross-service latency/errors |39| **SLO / error budget** | Define target (e.g. 99.9% / p95 latency); track budget; freeze risky changes when budget spent | Reliability is measured, not vibes |40| **On-call & runbook** | Who responds; written runbook per common failure (symptom → diagnosis → fix) | New engineer can resolve a page from the runbook |41| **Incident response** | Detect → mitigate → communicate (status page) → blameless postmortem with action items | Every incident produces a prevention task |42| **Deploys** | Progressive: canary or blue-green; automated rollback on error-rate spike; never big-bang prod | Bad deploy auto-reverts; no manual ftp |43| **Schema migrations** | Backward-compatible **expand→migrate→contract**; no locking DDL on hot tables; decoupled from app deploy; reversible | Migration can't break running old code; no long table lock |44| **Disaster recovery** | Define **RTO/RPO**; test region/AZ **failover**; backups ≠ DR; documented recovery procedure | Failover drill meets RTO/RPO |45| **Backups & restore** | Automated backups, **encrypted at rest + access-controlled**; **periodically test restore** (a backup never restored = no backup); offsite copy | Restore drill passes on a schedule |46| **Cert & secret rotation** | Automate TLS cert renewal; **alert on cert/secret expiry N days out**; rotate secrets on schedule | No expired-cert outage; rotation tested |47| **Queue / async health** | **Dead-letter queue** for poison messages past retry cap; alert on DLQ depth; monitor lag | Stuck messages isolated, not silently lost |48| **Database upkeep** | Index health, vacuum/analyze, slow-query review, connection-pool limits, archival of old rows | Queries stay fast as data grows |49| **Dependency patching** | Regular CVE scan + update cadence; pin + lockfile; test before bumping; track EOL/deprecated | No unpatched critical CVE lingering |50| **Log & data retention** | Log rotation + retention policy; **scheduled cleanup jobs** for expired/temp data; PII retention limits | Disks don't fill; data isn't hoarded forever |51| **Feature flags** | Decouple deploy from release; kill-switch for risky features; clean up stale flags | Can disable a feature without redeploy |52| **Cost monitoring** | Ongoing spend dashboards + budget alarms; catch runaway resources/queries | No surprise bill; cost tracked like latency |53| **Capacity & deprecation** | Periodic capacity review vs growth; planned sunset path for old APIs/versions with notice | Scale ahead of the wall; clean retirement |5455## Pre-Launch Ops Gate5657Do NOT call a live service production-ready until:58- [ ] Rate limiting + timeouts + retries (with backoff) on critical paths59- [ ] Graceful shutdown + health/readiness probes wired to the load balancer60- [ ] Monitoring dashboard + alerting to a real person live BEFORE launch61- [ ] Runbook for top failures; on-call owner named62- [ ] Progressive deploy + automated rollback tested63- [ ] Backups automated, encrypted, AND a restore drill passed64- [ ] DR: RTO/RPO defined; failover tested65- [ ] TLS cert auto-renew + expiry alerting; secret rotation set66- [ ] Distributed tracing wired; DLQ for async work67- [ ] Dependency CVE scan clean; patch cadence set68- [ ] Log retention + scheduled data-cleanup jobs configured69- [ ] Cost budget alarms set7071## Common Mistakes7273| Mistake | Fix |74|---|---|75| Rate limit only on login, not write APIs | Limit every abusable endpoint; return 429 + Retry-After |76| Retry without backoff/jitter | Retry storm DOSes your own backend — add backoff + jitter, cap attempts |77| Retrying non-idempotent writes | Add idempotency keys first, or don't retry |78| "We have backups" (never restored) | A backup never test-restored is not a backup — drill it |79| Alert on everything | Alert fatigue hides real fires — alert on SLO symptoms |80| Big-bang prod deploy | Canary/blue-green + auto-rollback |81| Caching with no invalidation plan | Define TTL + invalidation before adding cache |8283## Companion skills8485- Overall gate → `disciplined-delivery` (its "observable" + DoD reference this)86- Web build → `shipping-production-websites` (pillars 9/10/13 = this skill's depth)87- Security perimeter overlaps → `securing-applications` (rate-limit also a security control; WAF, A09 logging)
nhattrung0911