Test Ambiguity Detector

Test Ambiguity Detector

Compare a test file against its specification to identify where tests assume behavior that the spec doesn't explicitly define.

Usage: /test-ambiguity-detector file_backup checkpoint_1

Purpose

Tests should verify spec-defined behavior, not invent requirements. This skill finds cases where:

• Tests assert specific behavior the spec is silent about
• Tests assume implementation details not mandated by spec
• Tests expect particular error messages/codes not specified
• Tests require specific ordering/formatting beyond spec requirements

Step 1: Gather Context

Read the spec and test files for the specified problem/checkpoint:

problems/{problem}/checkpoint_N.md
problems/{problem}/tests/test_checkpoint_N.py
problems/{problem}/tests/conftest.py        # Fixtures and test setup

Also check for test data:

problems/{problem}/tests/data/checkpoint_N/  # Test case data (if present)

Step 2: Catalog Spec Requirements

For each section of the spec, extract:

1. Explicit requirements - "MUST", "SHALL", specific values, exact formats
2. Implicit requirements - Reasonable inferences from context
3. Undefined behaviors - What the spec does NOT address

Create a mental checklist:

• Input validation rules (if any specified)
• Output format requirements (exact vs flexible)
• Error handling expectations (if any specified)
• Edge cases addressed (explicitly or by example)
• Processing order requirements
• Default values specified

Step 3: Analyze Each Test

For each test function or test case:

A. Identify the Assertion

What specific behavior does this test check?

• Return values
• Output format/content
• Side effects
• Error conditions
• State changes

B. Find Spec Support

Can you point to a specific spec line that mandates this behavior?

SUPPORTED: Spec says "must return error code 1" → test checks returncode == 1 UNSUPPORTED: Spec says "must return non-zero on error" → test checks returncode == 1

C. Classify the Assumption

Step 4: Report Findings

Generate a report in this format:

# Ambiguity Report: {problem} {checkpoint}

## Summary
- Total tests analyzed: N
- Potentially ambiguous: N
- Likely spec issues: N

## Ambiguous Test Assumptions

### 1. `test_function_name` (line X)

**Assertion**: [What the test checks]

**Spec says**: [Quote or "NOTHING - spec is silent"]

**Risk**: [HIGH/MEDIUM/LOW]

**Analysis**: [Why this is problematic]

**Recommendation**:
- [ ] Clarify in spec: [suggested text]
- [ ] OR relax test: [how to make less specific]
- [ ] OR remove test: [if testing undefined behavior]

---

### 2. `test_another_function` (line Y)
...

## Spec Gaps Identified

List behaviors tested but not defined in spec:
1. [Behavior] - should spec define this?
2. ...

## Recommendations

### For Spec Authors
- [Suggested clarifications]

### For Test Authors
- [Suggested test modifications]

Common Ambiguity Patterns

Error Handling

• Ambiguous: Spec says "return error" but test expects specific code/message
• Check: Does spec define error codes? Error message format?

Ordering

• Ambiguous: Spec says "process items" but test expects specific order
• Check: Does spec mandate sorting? Stable ordering?

Whitespace/Formatting

• Ambiguous: Spec shows example output, test requires exact formatting
• Check: Does spec say "must match exactly" or just "must contain"?

Empty/Null Inputs

• Ambiguous: Test expects specific behavior for empty input
• Check: Does spec address empty/null cases?

Boundary Values

• Ambiguous: Test checks behavior at 0, max, or edge values
• Check: Does spec define boundary behavior?

Field Ordering in JSON/YAML

• Ambiguous: Test expects fields in specific order
• Check: Does spec mandate field ordering?

Case Sensitivity

• Ambiguous: Test expects case-sensitive/insensitive matching
• Check: Does spec specify case handling?

Red Flags in Tests

Watch for these patterns that often indicate ambiguity:

# Specific error code not in spec
assert result.returncode == 42

# Exact error message not in spec
assert "invalid format" in result.stderr

# Specific field ordering
assert json.dumps(output) == '{"a":1,"b":2}'

# Implicit sorting expectations
assert results == sorted(results)

# Default value assumptions
assert config.get("timeout") == 30

# Specific exception types
with pytest.raises(ValueError):

Example Analysis

Given spec text:

Return non-zero exit code on invalid input.

And test:

def test_invalid_input():
    result = subprocess.run(...)
    assert result.returncode == 1  # <-- AMBIGUOUS!

Finding: Test expects exit code 1, but spec only says "non-zero". Valid implementations could return 2, 127, or any non-zero value.

Recommendation: Either:

• Clarify spec: "Return exit code 1 on invalid input"
• Relax test: assert result.returncode != 0

Recommended Tools for Lenient Tests

Fix preference: Make tests lenient > Remove tests >> Update spec

Many ambiguity issues can be solved by using flexible comparison tools instead of exact matching.

deepdiff - Flexible Object Comparison

from deepdiff import DeepDiff

# Instead of exact matching:
assert actual == expected  # FAILS on order, float precision, extra fields

# Use deepdiff to ignore irrelevant differences:
diff = DeepDiff(expected, actual,
    ignore_order=True,           # List/dict order doesn't matter
    significant_digits=5,         # Float precision tolerance
    exclude_paths=["root['id']"]  # Ignore fields not in spec
)
assert not diff, f"Meaningful differences: {diff}"

jsonschema - Structure Validation

from jsonschema import validate

# Instead of checking exact values:
assert output == {"status": "ok", "count": 5}  # Too strict!

# Validate structure matches spec requirements:
schema = {
    "type": "object",
    "required": ["status", "count"],  # Only what spec requires
    "properties": {
        "status": {"enum": ["ok", "error"]},
        "count": {"type": "integer", "minimum": 0}
    }
}
validate(instance=output, schema=schema)  # Accepts any valid structure

Normalization - Handle Formatting Differences

import json

def normalize(obj):
    """Normalize for comparison: sort keys, strip whitespace, lowercase strings."""
    if isinstance(obj, dict):
        return {k: normalize(v) for k, v in sorted(obj.items())}
    if isinstance(obj, list):
        return sorted([normalize(x) for x in obj], key=lambda x: json.dumps(x, sort_keys=True))
    if isinstance(obj, str):
        return obj.strip().lower()
    return obj

# Instead of exact string/JSON comparison:
assert json.dumps(actual) == json.dumps(expected)  # Fails on key order!

# Normalize first:
assert normalize(actual) == normalize(expected)

Combined Approach

def flexible_assert(actual, expected, ignore_fields=None):
    """Compare with maximum leniency for spec compliance."""
    diff = DeepDiff(
        normalize(expected),
        normalize(actual),
        ignore_order=True,
        exclude_paths=ignore_fields or [],
        significant_digits=5,
        ignore_string_case=True
    )
    assert not diff, f"Spec-relevant differences: {diff}"

# Usage: Only fails on meaningful differences
flexible_assert(actual, expected, ignore_fields=["root['timestamp']"])

When to Use Each Tool

After Analysis

Summarize findings to the user with:

1. High-risk ambiguities that likely cause false failures
2. Spec gaps that should be addressed
3. Test modifications to reduce brittleness
4. Questions for spec authors to clarify intent

Converted and distributed by TomeVault — claim your Tome and manage your conversions.