yunseo-kim CI Triage
Use this skill when triaging CI failures, diagnosing flaky tests, or analyzing PR review comments. Provides a structured methodology for prioritizing and resolving CI issues efficiently.
Triage Prioritization
Investigate failures in this order — higher-priority categories block everything below:
- Build failures — Nothing else matters if the build doesn't compile
- Lint failures — Quick fixes, clear them first
- Type check failures — Often reveal real issues
- Test failures — Investigate after the above pass
- Review comments — Address after CI blockers are resolved
Failure Categories
Classify each failure to guide your response:
| Category | Description | Typical Action |
|---|---|---|
| Infrastructure/Transient | Network errors, 503s, service timeouts | Retry the job; not caused by code |
| Build | Compilation errors, missing dependencies | Fix imports, dependencies, or config |
| Lint | Formatting, style violations | Run formatter/linter autofix |
| Type Check | Type mismatches, missing declarations | Fix types at the source |
| Assertion | Wrong output, snapshot diffs, value mismatches | Debug the logic, update expectations |
| Timeout | Tests hanging, deadline exceeded | Find infinite loops, missing callbacks |
| Port Binding | EADDRINUSE, port conflicts | Fix test isolation, use dynamic ports |
| Routing/SSR | Wrong status codes, dynamic params unresolved | Check route config and handlers |
| Source Maps | Wrong file paths, incorrect line numbers | Check bundler source map settings |
| CLI Output | Missing warnings, wrong log order | Check stdio handling and ordering |
Failure Handling Rules
- Investigate each failing job as if it is caused by your changes.
- Do NOT assume flakiness by default.
- Only classify as flaky if the same test fails on
main/masterwith no related changes. - Check the last 3 main branch CI runs for the same failure before dismissing.
- If a CI run includes a "Known Flaky Tests" section, use it as historical context, NOT as automatic dismissal.
Common Failure Patterns
- Rust check / build failures: Run
cargo fmt -- --check, fix withcargo fmt. - Lint / formatting failures: Run the project formatter (e.g.,
prettier --write,eslint --fix). - Test failures: Run the exact failing test file locally, matching the CI job's dev vs production mode.
Local Reproduction
Match CI Environment
Mirror the CI job's environment variables and mode when reproducing locally:
- Read the CI job config — Extract env vars, test runner flags, and build mode from the job output.
- Set matching env vars — Export all relevant variables before running tests.
- Match the build mode — Dev vs production, bundler variant (webpack/turbopack/vite/esbuild/etc.).
# Example: capture output once, analyze multiple times
CI_ENV_VAR=value test-command test/path/to/test.ts > /tmp/test-output.log 2>&1
grep "FAIL\|Error" /tmp/test-output.log
grep -A5 "Error:" /tmp/test-output.log
tail -5 /tmp/test-output.log
Isolation Rule
When validating module-resolution, import path, or dependency fixes, run the full build without skip-isolation flags. Shortcuts that bypass packaging or bundling can hide the real issue.
One-Run Log Analysis
Capture the full test output once, then analyze without re-running:
# Capture
HEADLESS=true test-runner test/path/to/test.ts > /tmp/test-output.log 2>&1
# Analyze
grep "FAIL\|Error\|●" /tmp/test-output.log # Find failures
grep -A5 "Error:" /tmp/test-output.log # Error context
tail -5 /tmp/test-output.log # Exit status
Structured CI Log Analysis
Extracting Test Failures
For each failing CI job, extract structured data:
- TEST FILE — Full path (e.g.,
test/integration/auth/login.test.ts) - TEST NAME — The specific test case name
- JOB TYPE — CI job identifier (e.g.,
unit-tests,e2e-chrome,integration-node18) - EXPECTED — Exact expected value from the assertion
- RECEIVED — Exact received value from the assertion
- CATEGORY — One of: assertion | timeout | build | infrastructure | lint | type-check
- ROOT CAUSE — One-sentence hypothesis
- LOG FILE — Source log that led to the conclusion
Deduplication
Before summarizing:
- Group all failures by test file path, not by CI job name.
- If multiple jobs fail the same test file, count the jobs but report once.
- Identify systemic issues — the same test failing across many job variants indicates a real bug, not flakiness.
Parallel Analysis
For large CI suites with many failing jobs, parallelize the analysis:
- Batch 3-4 job logs per analysis pass (limit concurrency to avoid rate limits).
- Use a structured extraction template for each batch:
Analyze CI results for these jobs: [job-log-1] [job-log-2]
For each failing test, extract:
1. TEST FILE: (full path)
2. TEST NAME: (the specific test case name)
3. JOB TYPE: (the kind of job)
4. EXPECTED: (exact expected value from assertion)
5. RECEIVED: (exact received value from assertion)
6. CATEGORY: (assertion|timeout|build|infrastructure|lint|type-check)
7. ROOT CAUSE: (one sentence hypothesis)
8. LOG FILE: (analyzed log file that led to conclusion)
Return structured findings grouped by TEST FILE, not by job.
Also extract other failures not related to tests.
Identify if they are likely transient.
- Merge and deduplicate across batches.
PR Review Comment Analysis
Extracting Review Feedback
For each review thread or comment, extract:
- FILE — The file path being reviewed
- REVIEWER — Who left the comment
- STATUS — Open/Resolved (threads) or Approved/Changes Requested/Commented (reviews)
- TYPE — One of: code-style | bug | design | question | suggestion | nitpick | blocker
- SUMMARY — One-sentence summary of the feedback
- ACTION REQUIRED — Yes/No — does this require changes?
- PRIORITY — High (changes requested, blocker), Medium (open suggestion), Low (resolved, nitpick)
Extraction template:
Analyze PR review comments from these files: [review-data]
For each review thread/comment, extract:
1. FILE: (the file path being reviewed)
2. REVIEWER: (who left the comment)
3. STATUS: (Open/Resolved for threads, APPROVED/CHANGES_REQUESTED/COMMENTED for reviews)
4. TYPE: (code-style|bug|design|question|suggestion|nitpick|blocker)
5. SUMMARY: (one sentence summary of the feedback)
6. ACTION REQUIRED: (yes/no)
7. PRIORITY: (high if CHANGES_REQUESTED or blocker, medium if open suggestion, low if resolved or nitpick)
Return findings grouped by file path.
Prioritize review findings:
- Blockers — Must resolve before merge
- Changes Requested — Reviewer explicitly wants changes
- Suggestions — Worth considering but not blocking
- Nitpicks — Style preferences, optional
Summary Output Template
After analysis, produce a summary table:
| # | Test / Issue | Category | Jobs Affected | Priority | Fix Recommendation |
|---|-------------|----------|---------------|----------|--------------------|
| 1 | path/to/test.ts - "test name" | assertion | 3/8 jobs | HIGH | Fix expected value |
| 2 | Build step - missing dep | build | all jobs | HIGH | Add dependency |
| 3 | path/to/flaky.ts - "timeout" | timeout | 1/8 jobs | LOW | Known flaky |
Recommend fixes by priority: HIGH > MEDIUM > LOW.