Rca:ci

RCA-CI: Root Cause Analysis from CI Logs

Systematic root cause analysis when you only have access to CI logs and artifacts.

rca:ci vs rca:hypershift

When to Use

• CI pipeline failed
• No HyperShift cluster available
• Need to understand failure before deciding on fix
• Before using tdd:ci to iterate on fixes

Auto-approved: gh commands for downloading CI logs and artifacts are auto-approved. Analysis output is saved to /tmp/kagenti/rca/.

RCA Workflow

flowchart TD
    START(["/rca:ci"]) --> P1["Phase 1: Gather"]:::rca
    P1 --> P2["Phase 2: Isolate"]:::rca
    P2 --> P3["Phase 3: Hypothesize"]:::rca
    P3 --> P4["Phase 4: Verify"]:::rca
    P4 --> RESULT{"Conclusive?"}
    RESULT -->|Yes| P5["Phase 5: Document"]:::rca
    RESULT -->|No| ESC["Escalate to rca:hypershift"]:::rca
    P5 --> TDD["tdd:ci"]:::tdd

    classDef rca fill:#FF5722,stroke:#333,color:white
    classDef tdd fill:#4CAF50,stroke:#333,color:white

Follow this diagram as the workflow.

Phase 1: Gather CI Artifacts

Create working directory for analysis:

mkdir -p /tmp/kagenti/rca

# Get failed run ID
gh run list --status failure --limit 5

# Download logs to working directory
gh run view <run-id> --log-failed > /tmp/kagenti/rca/failed-log.txt

# View in browser for full context
gh run view <run-id> --web

# Download all artifacts to working directory
gh run download <run-id> -D /tmp/kagenti/rca/artifacts

Phase 2: Isolate the Failure

Find First Error

# Search for common error patterns in downloaded logs
grep -r "error\|Error\|ERROR\|failed\|Failed\|FAILED" ./*.txt | head -20

# Find assertion failures
grep -r "AssertionError\|assert\|FAILED" ./*.txt

# Find the first occurrence
grep -rn "error" ./*.txt | sort -t: -k2 -n | head -10

Error Chain Analysis

Work backwards from the failure:

1. What test failed?
2. What assertion failed?
3. What was the actual vs expected value?
4. What API/operation produced the wrong result?
5. What component is responsible?

Phase 3: Hypothesize Causes

Common CI Failure Categories

Hypothesis Template

## Hypothesis 1: [Brief description]
- **Likelihood**: High/Medium/Low
- **Evidence needed**: [What to look for]
- **Found**: [Yes/No/Partial]
- **Conclusion**: [Confirmed/Eliminated/Inconclusive]

Phase 4: Verify with Evidence

Search Patterns

# Auth issues
grep -i "oauth\|token\|401\|403\|unauthorized" logs/*.txt

# Timing issues
grep -i "timeout\|timed out\|deadline\|retry" logs/*.txt

# Connection issues
grep -i "connection\|refused\|unreachable\|dns" logs/*.txt

# Resource issues
grep -i "oom\|memory\|evict\|limit" logs/*.txt

Cross-Reference

• Compare with last successful run
• Check if same test passed before
• Look for recent code changes in affected area

Phase 5: Document Findings

## Root Cause Analysis

**Failure**: [Test name / description]
**Run ID**: [gh run id]

### Root Cause
[Clear statement of what caused the failure]

### Evidence
1. [Log line / artifact showing the issue]
2. [Supporting evidence]

### Contributing Factors
- [Any secondary causes]

### Fix
[Proposed solution]

### Prevention
[How to prevent recurrence]

Escalation to rca:hypershift

Escalate when:

• Logs are insufficient to determine root cause
• Need to inspect live state (secrets, configs, pod status)
• Need to reproduce with debugging enabled
• Multiple hypotheses remain after log analysis

rca:ci inconclusive? → Create cluster → rca:hypershift

Quick Reference

Related Skills

• rca:hypershift - RCA with live cluster access
• tdd:ci - Fix iteration after RCA
• superpowers:systematic-debugging - General debugging approach