Test Analyzer

Test Analyzer Skill

Purpose

Analyze CTRF test reports to identify failure patterns, suggest fixes, and provide insights into test health. Uses deterministic jq parsing for efficient, token-friendly analysis.

When to Use

Call this skill when:

• Tests have failed and user wants analysis
• User asks "what tests failed?", "analyze test failures", or "why did tests fail?"
• User wants to understand test failure patterns or trends
• User asks about slow tests, flaky tests, or test coverage

Quick Reference

Using ctrf-utils.sh:

ctrf-utils.sh summary <file>           # Show pass/fail summary
ctrf-utils.sh failures <file...>       # List failed test names
ctrf-utils.sh failures-detail <file>   # Show failures with error messages
ctrf-utils.sh flaky <file>             # Show tests with retry attempts
ctrf-utils.sh slowest <N> <file>       # Show N slowest tests

Direct jq commands:

jq '.results.summary' report.ctrf.json
jq -r '.results.tests[] | select(.status == "failed") | .name' report.ctrf.json

CTRF Schema Overview

CTRF reports have this structure:

{
  "results": {
    "tool": { "name": "vitest" },
    "summary": { "tests": 50, "passed": 48, "failed": 2, "skipped": 0 },
    "tests": [
      {
        "name": "should validate user input",
        "status": "failed",
        "duration": 156,
        "suite": "UserForm",
        "message": "Expected true, received false",
        "filePath": "src/components/UserForm.test.ts",
        "line": 23
      }
    ]
  }
}

Required fields per test: name, status, duration Full schema: ctrf-schema.json

Essential jq Examples (~10 most common)

1. Summary statistics

jq '.results.summary' /workspace/artifacts/test/vitest.ctrf.json

2. One-line summary

jq -r '.results.summary | "Tests: \(.tests) | Passed: \(.passed) | Failed: \(.failed)"' /workspace/artifacts/test/vitest.ctrf.json

3. List failed test names

jq -r '.results.tests[] | select(.status == "failed") | .name' /workspace/artifacts/test/*.ctrf.json

4. Failed tests with error messages

jq -r '.results.tests[] | select(.status == "failed") | "❌ \(.name): \(.message // "No message")"' /workspace/artifacts/test/*.ctrf.json

5. Group failures by error pattern

jq '[.results.tests[] | select(.status == "failed")] | group_by(.message) | map({error: .[0].message, count: length, tests: map(.name)}) | sort_by(.count) | reverse' /workspace/artifacts/test/*.ctrf.json

6. Slowest 10 tests

jq '.results.tests | sort_by(.duration) | reverse | limit(10; .[]) | {name, duration}' /workspace/artifacts/test/vitest.ctrf.json

7. Flaky tests (with retries)

jq -r '.results.tests[] | select(.retries > 0 or .flaky == true) | "⚠️ \(.name) (retries: \(.retries // 0))"' /workspace/artifacts/test/*.ctrf.json

8. Extract file paths for failed tests

jq -r '[.results.tests[] | select(.status == "failed") | .filePath] | unique | .[]' /workspace/artifacts/test/vitest.ctrf.json

9. Cross-project failure summary

jq -s '{vitest: .[0].results.summary.failed, pytest: .[1].results.summary.failed, total: ((.[0].results.summary.failed // 0) + (.[1].results.summary.failed // 0))}' /workspace/artifacts/test/vitest.ctrf.json /workspace/artifacts/test/pytest.ctrf.json

10. AI-friendly compact summary

jq -c '{summary: .results.summary, failures: [.results.tests[] | select(.status == "failed") | {name, suite, message, file: "\(.filePath):\(.line)"}]}' /workspace/artifacts/test/*.ctrf.json

More jq Examples

For comprehensive examples (24+), see: jq-examples-comprehensive.md

Scripts Reference

Implementation scripts:

• ctrf-utils.sh - CTRF utilities for deterministic parsing (used by skill)
• references/run-silent.sh - Silent-on-success wrapper pattern (reference)

Research Background

Context and research behind this approach:

• references/2026-01-15-context-efficient-backpressure-blogpost.md
• references/2026-01-15-test-output-strategies-ai-agents-claude.md
• references/2026-01-15-test-output-strategies-ai-agents-gemini.md
• references/2026-01-15-research-test-output-summarization-chatgpt.md

Skill Behavior

When invoked:

1. Locate CTRF reports (common locations: artifacts/test/, test-results/, project root)

2. Get summary:

   jq -r '.results.summary | "Tests: \(.tests) | Passed: \(.passed) | Failed: \(.failed)"' *.ctrf.json
   

3. If failures exist, list them:

   jq -r '.results.tests[] | select(.status == "failed") | "❌ \(.name): \(.message // "No message")"' *.ctrf.json
   

4. Analyze patterns (group similar errors):

   jq '[.results.tests[] | select(.status == "failed")] | group_by(.message) | map({error: .[0].message, count: length})' *.ctrf.json
   

5. Read source files for failed tests (extract file paths with jq, then read)

6. Suggest fixes based on patterns (group similar errors, identify root causes)