1---2name: troubleshooting3description: General debugging and troubleshooting patterns for slow APIs, deployment rollback, connection issues, OOMKilled pods, CrashLoopBackOff, and debugging principles. Covers root cause analysis, incident triage workflows, error stack trace analysis, pod logs inspection, metrics anomaly detection, alert investigation, and systematic debugging methodologies. Use when diagnosing errors, performing root cause analysis on production incidents, investigating alert anomalies, or triaging deployment failures.4license: MIT5---67# Troubleshooting Rules89## 1. Slow API Response1011### Bottleneck Identification12131. **Identify the bottleneck layer**: Controller → Service → Repository → External API142. **Check database queries**: Enable query logging or check slow query log153. **Check N+1 queries**: Look for repeated similar queries in logs164. **Check external API calls**: Measure response time of downstream services175. **Check thread/connection pool exhaustion**: Monitor active thread count and pool usage1819### Common Causes and Fixes2021| Cause | Diagnosis | Fix |22| ---------------------------- | ---------------------------------- | ------------------------------------ |23| N+1 queries | Multiple similar SQL in logs | Use join fetch or batch loading |24| Missing index | Slow query log, full table scan | Add appropriate index |25| Large payload serialization | High CPU during response | Use pagination, field selection |26| Synchronous external API | Thread blocked waiting | Use async call or timeout |27| Connection pool exhaustion | Requests queued, pool warnings | Increase pool size or optimize query |28| No caching | Same expensive query repeated | Add cache layer |2930---3132## 2. Deployment Rollback3334### Rollback Criteria (any of the following)3536- Error rate exceeds baseline by 5x or more37- P99 latency exceeds SLA threshold38- Critical business flow is broken (login, payment, etc.)39- Crash loop detected in pods40- Data corruption observed4142### Kubernetes Rollback Procedure4344```bash45# Rollback to previous revision46kubectl rollout undo deployment/<app-name> -n <namespace>4748# Verify rollback status49kubectl rollout status deployment/<app-name> -n <namespace>5051# Check pod health after rollback52kubectl get pods -n <namespace> -l app=<app-name>53```5455### Post-Rollback Checklist5657- [ ] Verify application health and metrics returned to normal58- [ ] Notify team of rollback and reason59- [ ] Create incident report if applicable60- [ ] Root-cause analysis before re-deploying the change61- [ ] Add test coverage for the failure scenario6263---6465## 3. Connection and Network Issues6667> See [references/database-connection-troubleshooting.md](references/database-connection-troubleshooting.md) for detailed HikariCP pool sizing, leak detection, and connection pool troubleshooting patterns.6869### Database Connection7071| Symptom | Check | Fix |72| ---------------------- | ---------------------------- | ------------------------------------- |73| `Connection refused` | Is DB running? Port open? | Verify DB status and network/firewall |74| `Connection timed out` | Network latency, firewall | Check security group, VPC peering |75| `Too many connections` | Pool size vs DB max conns | Tune pool size, check for leaks |76| `Connection reset` | Idle conn killed by proxy/LB | Set connection and idle timeouts |7778### External API Connection7980| Symptom | Check | Fix |81| ------------------------- | ------------------------ | ------------------------------------- |82| `ConnectTimeoutException` | Target reachable? | Verify DNS, firewall, service health |83| `ReadTimeoutException` | Response too slow | Increase timeout or optimize upstream |84| `SSLHandshakeException` | Certificate issue | Check cert validity, trust store |85| `429 Too Many Requests` | Rate limited | Implement backoff, request quota |8687---8889## 4. General Debugging Principles9091### Do9293- Start with logs — check ERROR and WARN levels first94- Reproduce the issue locally before investigating in production95- Use traceId to follow a request across services96- Check "what changed" — recent deployments, config changes, traffic spikes97- Narrow down the scope: which endpoint, which user, which time window9899### Do Not100101- Make changes to production without understanding the root cause102- Restart pods as the first response — investigate first103- Ignore intermittent errors — they often indicate resource exhaustion104- Debug with print statements — use structured logging105- Chase symptoms without identifying the root cause106107---108109## 5. Anti-Patterns110111- Applying fixes without understanding root cause112- No runbook for common failure scenarios113- Missing alerting for critical business flows114- No baseline metrics to compare against during incidents115- Skipping post-mortem after production incidents116117## 6. Related Skills118119- `observability`: System monitoring and alerting setup120- `logging`: Log-based problem diagnosis121- `incident-response`: Incident response processes122123## Additional References124125- For Kubernetes troubleshooting (OOMKilled, CrashLoopBackOff, Pod debugging), see [references/kubernetes-troubleshooting.md](references/kubernetes-troubleshooting.md)126- For JVM troubleshooting (heap dump, thread dump, GC analysis, diagnostic tools), see [references/jvm-troubleshooting.md](references/jvm-troubleshooting.md)127- For database connection and connection pool troubleshooting (HikariCP, leak detection), see [references/database-connection-troubleshooting.md](references/database-connection-troubleshooting.md)128- [Google SRE Book - Debugging](https://sre.google/sre-book/effective-troubleshooting/) - Effective troubleshooting methodology129- [Brendan Gregg's Systems Performance](https://www.brendangregg.com/systems-performance-2nd-edition-book.html) - Systems performance analysis130- For Spring Boot troubleshooting (startup failures, JVM OOM, HikariCP), see `spring-framework` skill — [references/troubleshooting.md](../spring-framework/references/troubleshooting.md)
iceflower