Deployment Workflow
Follow this procedure for deploying a service to production. This skill covers pre-flight checks, staging validation, production rollout, and post-deployment monitoring.
Prerequisites
- You must have the
run_teststool available. - Access to the deployment pipeline (CI/CD).
- The service must be on the
mainbranch with all tests passing.
Step 1 — Pre-flight Checks
- Verify tests pass — use the
run_teststool withsuite: "all".- All unit and integration tests must be green.
- If any test fails, stop — do not proceed with deployment.
- Check the changelog — confirm that
CHANGELOG.mdhas been updated for this release. - Review open issues — ensure no critical/blocker issues are tagged against this release milestone.
Step 2 — Version Tagging
Create a semantic version tag following SemVer:
| Change type | Version bump | Example |
|---|---|---|
| Breaking change | Major | v1.0.0 → v2.0.0 |
| New feature | Minor | v1.0.0 → v1.1.0 |
| Bug fix / patch | Patch | v1.0.0 → v1.0.1 |
git tag -a v<X.Y.Z> -m "Release v<X.Y.Z>"
git push origin v<X.Y.Z>
Step 3 — Deploy to Staging
- Trigger the staging deployment pipeline.
- Wait for the deployment to complete (typically 3–5 minutes).
- Run smoke tests against the staging environment:
- Health check endpoint returns
200 OK. - Key user flows work end-to-end.
- No error spikes in the staging log dashboard.
- Health check endpoint returns
If staging fails
- Rollback immediately using the previous stable tag.
- Investigate the failure, fix, and restart from Step 1.
Step 4 — Deploy to Production
⚠️ Only proceed after staging sign-off.
- Trigger the production deployment pipeline.
- Use a canary deployment strategy:
- Route 10% of traffic to the new version.
- Monitor error rates for 5 minutes.
- If error rate stays below 0.1%, proceed to full rollout.
- If error rate exceeds 0.1%, rollback immediately.
- Gradually increase traffic: 10% → 25% → 50% → 100%.
Step 5 — Post-Deployment Monitoring
For the first 30 minutes after full rollout, actively monitor:
- Error rates in the logging dashboard.
- Response latency (p50, p95, p99).
- CPU and memory utilisation of service pods.
- Customer-facing alerts and support channels.
If any anomaly is detected, trigger an immediate rollback and create a post-incident report.
Rollback Procedure
# Deploy the previous known-good tag
deploy --service <service-name> --tag <previous-tag> --environment production
Always notify the team in #deployments when a rollback occurs.