Validation Protocol

Fix Validation Protocol

Overview

Before claiming any fix works, follow this scientific validation protocol.

When to Use This Skill

• Claiming a bug fix is complete
• Verifying a code change works as intended
• Validating an implementation meets requirements
• Confirming a refactoring preserves functionality
• Testing that a feature behaves correctly

Core Principle

Success = Observing the intended behavior, not absence of errors.

A fix that "runs without failing" is not validated. A fix that demonstrates the specific expected outcome is validated.

The Protocol

Step 1: Reproduce the Failing State

Objective: Establish the broken baseline before attempting any fix.

Actions:

• Explicitly create or verify the broken condition exists
• Document the observable symptoms:
  • Exact error messages
  • Wrong values or outputs
  • Unexpected behavior
  • System state issues
• Confirm you can observe the failure consistently

Example:

# Create the broken state by running the relevant command or test
{test command from language manifest} -k test_broken_function

# Observe the failure
# Expected: Error or incorrect output
# Observed: Error or incorrect output confirmed

Step 2: Define Success Criteria

Objective: State what specific observable output indicates the fix worked.

Actions:

• Identify the specific behavior that proves success
• Define measurable, observable outcomes
• Distinguish success from non-success indicators:
  • Success = The fix's intended behavior is demonstrated
  • Success != Absence of errors
  • Success != Absence of warnings
  • Success != "It ran without failing"

Example:

Success Criteria:
- Function returns expected value: {"status": "processed", "count": 42}
- No exceptions raised
- Output matches test assertion exactly
- Performance within acceptable range (<100ms)

Step 3: Apply the Fix and Observe

Objective: Implement the fix and capture what actually happens.

Actions:

• Run the code with the fix applied
• Look for the specific success indicators defined in Step 2
• Document what actually happened (verbatim output, return values, behavior)
• Compare observed outcome against success criteria

Example:

# Run the fixed code
{test command from language manifest} -k test_fixed_function

# Observe the output
# Expected: All assertions pass with correct values
# Observed: All assertions pass with correct values

Step 4: Verify the Result

Objective: Confirm the broken state is now fixed and success criteria are met.

Actions:

• Check that the broken state no longer exists
  • Run the same reproduction steps from Step 1
  • Verify the failure no longer occurs
• Confirm the success criteria from Step 2 are satisfied
  • Each criterion must be met with evidence
  • Partial success is not success
• Document the verification evidence

Example:

Verification Results:

Step 1 Recheck:
- Reproduction steps no longer trigger failure
- Error message no longer appears

Step 2 Criteria Check:
- Function returns {"status": "processed", "count": 42}
- No exceptions raised
- Output matches test assertion
- Performance: 45ms (within <100ms requirement)

Conclusion: Fix verified. All success criteria met with evidence.

Common Anti-Patterns to Avoid

Anti-Pattern 1: Claiming Success Without Reproducing Failure

"I fixed the bug. The code runs now."

Problem: Without reproducing the failure, you don't know if the fix addresses the actual issue.

Correct Approach:

Step 1: Reproduced failure - function raised ValueError("Invalid input")
Step 2: Success = function returns valid output without error
Step 3: Applied fix - added input validation
Step 4: Verified - function now returns {"result": "valid"}, no ValueError

Anti-Pattern 2: Confusing "No Errors" with Success

"The tests pass now, so the fix works."

Problem: Tests passing means no exceptions, not necessarily correct behavior.

Correct Approach:

Step 2: Success = function processes 1000 records and returns count=1000
Step 3: Observed output: {"processed": 1000, "failed": 0}
Step 4: Verified count matches expected value exactly

Anti-Pattern 3: Skipping Verification

"I made the change. It should work now."

Problem: "Should work" is speculation, not verification.

Correct Approach:

Step 3: Applied fix
Step 4: Ran test suite - all 45 tests pass
Step 4: Manually tested edge case - correct behavior observed
Step 4: Checked logs - no error messages, expected INFO logs present

Anti-Pattern 4: Partial Verification

"The main case works, so the fix is complete."

Problem: Edge cases and boundary conditions may still be broken.

Correct Approach:

Step 2: Success criteria:
  - Normal input: returns expected output
  - Empty input: raises ValueError
  - Large input (10k records): completes within 5s
  - Invalid input: raises TypeError

Step 4: All criteria verified with evidence

Integration with Testing

This validation protocol complements but does not replace automated testing:

Automated Tests: Prevent regressions, verify expected behavior systematically

Validation Protocol: Ensures the specific fix addresses the specific problem observed

Use both:

1. Follow validation protocol to verify the fix
2. Add automated test to prevent regression
3. Run full test suite to ensure no new issues