DevOps

DevOps Rules

CI/CD Pipelines

• Fail fast: run linting and unit tests before expensive integration tests — saves time and compute
• Cache dependencies between runs — npm install on every build wastes minutes
• Pin action versions with SHA, not tags — actions/checkout@v3 can change, SHA is immutable
• Secrets in environment variables, never in code or logs — mask them in CI output
• Parallel jobs for independent steps — test, lint, and build can run simultaneously

Deployment Strategies

• Blue-green: run new version alongside old, switch traffic atomically — instant rollback by switching back
• Canary: route percentage of traffic to new version — catch issues before full rollout
• Rolling: update instances incrementally — balance between speed and risk
• Always have rollback plan before deploying — know exactly how to revert
• Deploy the same artifact to all environments — build once, promote through stages

Infrastructure as Code

• Version control all infrastructure — terraform, ansible, cloudformation in git
• Never apply changes without plan/diff review — terraform plan before apply
• State files contain secrets — store remotely with encryption, never in git
• Modules for reusable components — don't copy-paste infrastructure definitions
• Separate environments with workspaces or directories — dev changes shouldn't affect prod

Containers

• One process per container — containers are not VMs
• Health checks are mandatory — orchestrators need them for routing and restarts
• Don't run as root — use non-root USER in Dockerfile
• Immutable images: config via environment, not baked in — same image in all environments
• Tag images with git SHA, not just latest — know exactly what's deployed

Secrets Management

• Never store secrets in environment files committed to git — use vault, sealed secrets, or CI secret storage
• Rotate secrets regularly — automation makes rotation painless
• Different secrets per environment — dev leak shouldn't compromise prod
• Audit secret access — know who accessed what and when
• Secrets in memory, not disk when possible — temp files persist longer than expected

Monitoring & Alerting

• Four golden signals: latency, traffic, errors, saturation — start here
• Alert on symptoms, not causes — "users seeing errors" not "CPU high"
• Every alert must be actionable — if you can't do anything, it's noise
• Dashboard per service with key metrics — one glance shows health
• Structured logs (JSON) for machine parsing — grep works, but queries are better

Reliability

• Define SLOs before building alerting — what does "healthy" mean for this service?
• Error budgets: some failures are acceptable — 99.9% means 8 hours downtime/year is OK
• Chaos engineering in staging — break things intentionally before prod breaks accidentally
• Runbooks for common incidents — 3am is not the time to figure out recovery steps
• Post-mortems without blame — focus on systems, not people

Common Mistakes

• SSH into prod to fix things — all changes through automation, or you'll forget what you did
• No staging environment — "works on my machine" doesn't mean works in prod
• Ignoring flaky tests — they erode trust in CI, either fix or delete
• Manual steps in deployment — if it's not automated, it'll be done wrong eventually
• Monitoring only happy paths — check error rates and edge cases too

Networking

• Internal services don't need public IPs — use private subnets, expose only load balancers
• TLS everywhere, including internal traffic — zero trust, even behind firewall
• DNS for service discovery — hardcoded IPs break when things move
• Load balancer health checks separate from app health — LB needs fast response, app health can be thorough
• Firewall default deny — explicitly allow what's needed, block everything else