Test Quality Checker

You are evaluating the quality of a test. Your task is to analyze the test and produce a JSON quality assessment.

Inputs

• Source file: {{source_file}}
• Test harness: {{test_harness}}
• Test name: {{test_name}}
• Build command (optional): {{build}}
• Test command with Coverage (optional): {{coverage}}

Step 1: Extract Test Information

Use the provided scripts to extract the test code and patterns:

# Extract assertions from the test
python scripts/assertion_analyzer.py extract {{test_harness}} --test {{test_name}} --json .deeptest/quality/assertions.json

# Extract the test code and patterns
python scripts/test_design_analyzer.py extract {{test_harness}} --test {{test_name}} --json .deeptest/quality/test_code.json

What the scripts extract (deterministic):

assertion_analyzer.py:

• All assertions with line numbers, code, type (equality/null_check/exception/etc.), and operands
• Summary flags: has_exception_handling, has_null_checks, has_equality_checks, has_boundary_checks
• Raw test code lines for additional analysis

test_design_analyzer.py:

• All numbers found (with line and context)
• All function calls found (with line and context)
• Language keywords found (global, static, extern, etc.)
• All comments found:
  • description_comment: The comment block immediately before the test (docstring, block comment, or consecutive line comments)
  • inline_comments: Single-line comments within the test (// or #)
  • block_comments: Multi-line /* */ comments within the test
• Raw test code lines for additional analysis

What you (the agent) must analyze:

The scripts provide raw data and deterministic pattern matches. You must interpret this data to determine:

• Assertion quality: Are the extracted assertions checking exact values? Are they comprehensive?
• Test structure: Looking at the raw code, identify setup/arrange, action/act, assertion/verify, cleanup phases
• Magic numbers: From the numbers found, which are problematic vs acceptable constants?
• External dependencies: From function calls, identify file I/O, network, database access
• Flakiness risks: From function calls, identify sleep, rand, time, etc.
• Design issues: From keywords and structure, identify global state, missing cleanup, etc.
• Comment-test conformity: Do the comments accurately describe what the test actually does?
  • Does the description comment match the actual test behavior?
  • Are inline comments accurate descriptions of the code?
  • If comments describe certain scenarios or expectations, are those actually tested?

Note: The raw code is included so you can extract any additional patterns or information not covered by the deterministic extraction.

Step 2: Measure Coverage

{{#if coverage}} Use the provided coverage commands:

{{build}}
{{coverage}}

{{else}} Use the coverage-analysis skill to simulate coverage:

Invoke coverage-analysis skill with:
- source_file: {{source_file}}
- test_file: {{test_harness}}
- test_case: {{test_name}}

{{/if}}

Collect:

• Lines covered in {{source_file}}
• Branches covered
• Total lines and branches in the target function

Step 3: Analyze and Score

Based on the extracted information, evaluate each dimension:

Assertion Quality (1-10)

Review the assertions from Step 1 and consider:

• Specificity: Do assertions check exact expected values or just non-null/truthy?
• Completeness: Are all observable outcomes (return values, side effects, state changes) verified?
• Relevance: Do assertions directly relate to the behavior being tested?
• Negative testing: Are error conditions and edge cases checked?

Coverage Differential (1-10)

Based on coverage from Step 2:

• What percentage of the target function's lines does this test cover?
• What percentage of branches are exercised?

Mutant Detection Ability (1-10)

Analyze the test's ability to detect code mutations:

• Review each assertion and determine what code changes it would catch
• Consider: arithmetic operators, relational operators, logical operators, constants, return values
• Estimate what percentage of realistic mutants this test would kill

Test Design Quality (1-10)

Based on structure from Step 1:

• Structure: Is there clear Arrange-Act-Assert separation?
• Naming: Is the test name descriptive of scenario and expectation?
• Maintainability: Are there magic numbers? Is cleanup present where needed?
• Robustness: Are there flakiness risks (timing, randomness, external dependencies)?
• Comment-test conformity: Do comments accurately describe what the test does?
  • If the description comment says "test that X throws exception", does the test actually verify that?
  • If inline comments describe setup or assertions, is that accurate?
  • Are there misleading or outdated comments?

Step 4: Generate JSON Output

Produce the final assessment as JSON with equal weights for all four measurements:

{
  "test_file": "{{test_harness}}",
  "test_name": "{{test_name}}",
  "source_file": "{{source_file}}",
  "assertion_quality": {
    "score": <1-10>,
    "specificity": <1-10>,
    "completeness": <1-10>,
    "relevance": <1-10>,
    "negative_testing": <1-10>,
    "issues": [
      {"type": "<issue_type>", "description": "<description>", "line": <line_number>}
    ]
  },
  "mutant_detection_ability": {
    "score": <1-10>,
    "estimated_kill_rate": <0.0-1.0>,
    "detectable_mutant_types": ["<type1>", "<type2>"],
    "undetectable_mutant_types": ["<type1>", "<type2>"],
    "analysis": "<explanation of detection capability>"
  },
  "coverage_differential": {
    "score": <1-10>,
    "lines_covered": <count>,
    "total_lines": <count>,
    "branches_covered": <count>,
    "total_branches": <count>,
    "line_coverage_percent": <0.0-100.0>,
    "branch_coverage_percent": <0.0-100.0>
  },
  "test_design_quality": {
    "score": <1-10>,
    "structure": <1-10>,
    "naming": <1-10>,
    "maintainability": <1-10>,
    "robustness": <1-10>,
    "comment_conformity": <1-10>,
    "issues": [
      {"type": "<issue_type>", "description": "<description>"}
    ],
    "comment_analysis": {
      "has_description": <true/false>,
      "description_matches_behavior": <true/false>,
      "misleading_comments": [
        {"line": <line_number>, "comment": "<comment text>", "issue": "<why misleading>"}
      ]
    }
  },
  "overall_assessment": {
    "score": <1-10>,
    "grade": "<A/B/C/D/F>",
    "summary": "<one sentence summary>",
    "strengths": ["<strength1>", "<strength2>"],
    "weaknesses": ["<weakness1>", "<weakness2>"],
    "recommendations": [
      {"priority": "<high/medium/low>", "action": "<specific improvement>"}
    ]
  }
}

Overall Score Calculation (Equal Weights)

overall_score = (
    assertion_quality.score +
    mutant_detection_ability.score +
    coverage_differential.score +
    test_design_quality.score
) / 4

Grade Mapping

• A: 9-10 (Excellent)
• B: 7-8 (Good)
• C: 5-6 (Acceptable)
• D: 3-4 (Poor)
• F: 1-2 (Failing)

Save the JSON output to .deeptest/quality_reports/{{test_name}}_quality.json

Converted and distributed by TomeVault — claim your Tome and manage your conversions.