# Payment 503 Spike

> Payment Service HTTP 503 Spike After Release

- Skill: `tommy-yw/payment-503-spike` (Agent Skill)
- Install (CLI): `npx skillmds@latest add tommy-yw/payment-503-spike`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tommy-yw/payment-503-spike/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tommy-yw (https://skillmd.com/u/tommy-yw)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tommy-yw/payment-503-spike

---

# Payment Service HTTP 503 Spike After Release

Use this skill when `payment-service` starts returning HTTP 503 shortly after a release, especially after version `v2.3.1`.

## Goal
Find whether the new release caused a database connection pool regression, then propose a safe rollback path.

## Evidence to collect
1. `prom_top_anomalies(service="payment-service")`
   - Look for `http_503_rate`, p95 latency, and request volume changes.
2. `loki_query(service="payment-service", query="connection pool exhausted")`
   - Look for database pool exhaustion or timeout messages.
3. `trace_search(service="payment-service", error_only=true)`
   - Look for slow or failed spans to `mysql-payment`.
4. `recent_deploys(service="payment-service")`
   - Check whether `v2.3.1` was deployed shortly before the HTTP 503 increase.
5. `incident_rca_guard(...)`
   - Confirm the evidence is consistent before producing the final RCA.
6. `action_policy_guard(...)`
   - Generate policy-checked actions.

## Decision logic
A release regression is likely when all of these are true:

- `payment-service` has a recent deployment such as `v2.3.1`.
- HTTP 503 rate increases after the deployment.
- Logs include `connection pool exhausted` or similar DB pool messages.
- Trace evidence points to `mysql-payment` latency or failed spans.

## Recommended action
If the above evidence is present, propose:

- `rollback_canary(service="payment-service", target_revision="v2.3.0", dry_run=true)` first.
- If the operator approves, call `rollback_canary(..., dry_run=false, approval_id="...")`.
- Then call `verify_recovery(service="payment-service")`.

## Safety rules

- Never execute rollback without approval.
- Always create a checkpoint before destructive execution.
- Keep raw logs and full traces out of stable memory.
- Final answer must cite evidence IDs.

## Final answer template

- Root cause: ...
- Evidence: `ev_metric_http_503_rate`, `ev_log_pool_exhausted`, `ev_trace_mysql_latency`, `ev_deploy_v231`
- Recommended action: rollback from `v2.3.1` to `v2.3.0` after approval.
- Verification: check HTTP 503 rate and p95 latency for 10 minutes.