Test Falsify

Test Falsifiability Verifier

Verify that tests fail when their declared bugs exist. A test that passes with the bug present is worthless.

When to Run

• After creating/modifying E2E or integration tests
• Before reporting test results in a PR
• When /test-falsify is explicitly invoked
• When a test looks suspiciously permissive

Execution

Phase 1: Extract Intent

From the target test file, extract:

1. Bug declarations: keywords Bug, detect, regression, fix in docstrings/comments.
2. Test name intent: test_long_conversation_natural_ending -> validates "natural ending".
3. User requirements: check conversation history for the original test request.

Evaluate each assertion against three checks:

Check 1: Falsifiability (CRITICAL)

Would this assertion fail if the declared bug exists?

Check 2: Uniformity Detection

Does the test verify that different inputs produce different outputs?

Check 3: Domain Consistency

Do test results contradict domain common sense?

Refer to references/domain-checklist.md for domain-specific checks.

Provide concrete assertion additions:

# Variance check: different answers produce different turn counts
results = [run(verbose_answers), run(brief_answers), run(mixed_answers)]
turn_counts = [r.turn_count for r in results]
assert max(turn_counts) - min(turn_counts) >= 3, "Fixed turn count = ignoring answers"

# Upper bound: reject known-bad fixed value
assert result.turn_count < 35, "37-turn fixed pattern detected"

# Uniformity rejection: not all models produce identical results
followup_counts = [r.followup_count for r in model_results]
assert len(set(followup_counts)) > 1, "All models identical = mechanical"

# Domain-specific: detailed answers should end sooner
verbose = run(very_detailed_answers)
brief = run(minimal_answers)
assert verbose.turn_count < brief.turn_count, "Not evaluating answer quality"

## Falsifiability Review

### Target: `path/to/test_file.py`

### Intent-Assertion Alignment
| # | Declared Intent | Assertion | Fails with bug? | Verdict |
|---|----------------|-----------|-----------------|---------|
| 1 | Bug 3: 37-turn fix | turn_count >= 18 | No (37>=18) | INVALID |
| 2 | Natural ending | has_farewell() | Yes | WEAK |

### Uniformity Check
- Multiple input patterns: YES / **NO**
- Variance verification: YES / **NO**

### Domain Consistency
- [Any domain-specific anomalies]

### Recommended Fixes
1. [Specific assertion to add]
2. [Test case to add]

References

• references/domain-checklist.md -- Domain-specific test consistency checks