Test Case Reducer

Test Case Reducer

Automatically reduce bug-triggering test cases to minimal form while guaranteeing the reduced test still reproduces the same failure.

Core Concept

Test case reduction works by:

1. Identify: Parse the test case and identify reducible elements
2. Remove: Systematically try removing elements (lines, statements, inputs)
3. Validate: Run the test after each removal to verify failure is preserved
4. Iterate: Continue until no further reduction is possible

Workflow

1. Prepare the Test Case

Ensure you have:

• A test case that consistently reproduces the failure
• A way to run the test (command, script, test framework)
• A clear failure condition (oracle)

2. Define the Oracle

The oracle determines if the test still exhibits the expected failure:

Exit code:

• Test fails with specific exit code (e.g., non-zero)
• Example: --oracle exit_code --expected 1

Exception type:

• Test raises specific exception
• Example: --oracle exception --expected ValueError

Output pattern:

• Test output contains specific text
• Example: --oracle output_pattern --expected "division by zero"

Assertion:

• Test fails on specific assertion
• Example: --oracle assertion --expected AssertionError

3. Choose Reduction Strategy

Aggressive (fastest, smallest result):

• Binary search reduction first
• Delta debugging second
• Greedy line-by-line cleanup
• Use when: Test execution is fast, minimal size is critical

Balanced (recommended):

• Delta debugging only
• Good balance of speed and quality
• Use when: Standard reduction needs

Conservative (safest, most readable):

• Greedy line-by-line only
• Preserves more context
• Use when: Readability matters, test is already small

4. Run the Reduction

Using the provided script:

python scripts/reduce_test.py test_file.py \
  --command "python test_file.py" \
  --oracle exit_code \
  --expected 1 \
  --strategy balanced

Or manually apply reduction algorithms (see references/algorithms.md).

5. Review the Result

Examine the reduced test case:

• Verify it still reproduces the failure
• Check that it's understandable
• Ensure no critical context was lost

Quick Start Examples

Example 1: Python Unit Test

Original test (15 lines):

import unittest
from mymodule import Calculator

class TestCalculator(unittest.TestCase):
    def setUp(self):
        self.calc = Calculator()
        self.test_data = [1, 2, 3, 4, 5]

    def test_divide(self):
        result = self.calc.divide(10, 2)
        self.assertEqual(result, 5)

        result = self.calc.divide(10, 0)  # This fails
        self.assertEqual(result, 0)

if __name__ == '__main__':
    unittest.main()

Reduction command:

python scripts/reduce_test.py test_calc.py \
  --command "python test_calc.py" \
  --oracle exception \
  --expected "ZeroDivisionError"

Reduced test (3 lines):

from mymodule import Calculator
Calculator().divide(10, 0)

Example 2: JavaScript Integration Test

Original test (20 lines):

const request = require('supertest');
const app = require('../app');

describe('API Tests', () => {
    beforeEach(() => {
        // Setup database
        db.reset();
        db.seed();
    });

    it('should handle invalid user ID', async () => {
        const response = await request(app)
            .get('/api/users/invalid')
            .set('Authorization', 'Bearer token')
            .expect(400);

        expect(response.body.error).toBe('Invalid user ID');
    });
});

Reduced test (4 lines):

const request = require('supertest');
const app = require('../app');

request(app).get('/api/users/invalid').expect(400);

Example 3: Input File Reduction

Original input (100 lines of JSON):

{
  "users": [...100 user objects...],
  "settings": {...many settings...},
  "data": [...large data array...]
}

Reduced input (5 lines):

{
  "users": [{"id": 42, "name": "Bob"}]
}

Using the Reduction Script

Basic Usage

python scripts/reduce_test.py <test_file> \
  --command "<command to run test>" \
  --oracle <oracle_type> \
  --expected <expected_value>

Parameters

Required:

• test_file: Path to the test file to reduce
• --command: Command to execute the test (e.g., python test.py, npm test)
• --oracle: Type of failure oracle (exit_code, exception, output_pattern, assertion)
• --expected: Expected value for the oracle

Optional:

• --strategy: Reduction strategy (aggressive, balanced, conservative)
• --timeout: Timeout in seconds for test execution

Examples

Python test with exception:

python scripts/reduce_test.py test.py \
  --command "python test.py" \
  --oracle exception \
  --expected "ValueError"

JavaScript test with exit code:

python scripts/reduce_test.py test.js \
  --command "node test.js" \
  --oracle exit_code \
  --expected 1 \
  --strategy aggressive

Java test with timeout:

python scripts/reduce_test.py Test.java \
  --command "javac Test.java && java Test" \
  --oracle output_pattern \
  --expected "NullPointerException" \
  --timeout 10

Manual Reduction Workflow

When not using the script, follow this process:

1. Verify Original Failure

# Run original test
python test.py
# Verify it fails as expected

2. Try Removing Large Chunks

Remove half the test:

# Original
line1
line2
line3
line4

# Try first half
line1
line2

# Or second half
line3
line4

3. Apply Delta Debugging

See references/algorithms.md for detailed algorithm.

4. Clean Up Line by Line

Remove one line at a time, testing after each removal.

5. Verify Final Result

Ensure reduced test still fails with same error.

Language-Specific Guidance

For detailed language-specific reduction techniques, see references/language-specific.md.

Python:

• Remove unused imports
• Simplify test setup
• Reduce assertions to minimum

JavaScript:

• Remove describe blocks if possible
• Simplify mock setup
• Reduce to single assertion

Java:

• Remove @Before/@After if not needed
• Simplify test class structure
• Reduce verbose assertions

C/C++:

• Remove unnecessary includes
• Simplify test fixtures
• Reduce to single EXPECT/ASSERT

Common Scenarios

Scenario 1: Large Integration Test

Problem: Integration test with 200 lines fails intermittently

Solution:

1. Run test multiple times to ensure consistent failure
2. Use aggressive strategy for fast reduction
3. Reduce to minimal API calls that trigger failure

Scenario 2: Complex Input File

Problem: 10MB JSON file causes parser to crash

Solution:

1. Use binary search to find problematic section
2. Apply delta debugging to reduce section
3. Result: Small JSON snippet that triggers crash

Scenario 3: Verbose Unit Test

Problem: Unit test with extensive setup fails on one assertion

Solution:

1. Use conservative strategy to preserve readability
2. Remove setup code not needed for failure
3. Keep minimal context for understanding

Best Practices

Before Reduction

• Verify determinism: Ensure test fails consistently
• Backup original: Keep copy of original test
• Document failure: Note exact error message and conditions

During Reduction

• Start aggressive: Use aggressive strategy first, then refine
• Validate frequently: Check each reduction preserves failure
• Watch for side effects: Be careful with tests that modify state
• Set timeouts: Prevent hanging on infinite loops

After Reduction

• Verify manually: Run reduced test several times
• Check readability: Ensure reduced test is understandable
• Add comments: Document what the test is checking
• Create regression test: Use reduced test as regression test

Troubleshooting

Reduction removes too much:

• Use conservative strategy
• Preserve specific elements manually
• Check oracle is correct

Test becomes non-deterministic:

• Original test may have race condition
• Run multiple times to verify
• Consider recording non-deterministic inputs

Reduction is too slow:

• Use aggressive strategy
• Increase timeout
• Simplify test command

Syntax errors after reduction:

• Use language-aware reduction
• Preserve structural elements
• Validate syntax after each step

References

• algorithms.md: Detailed explanation of delta debugging, binary search, and other reduction algorithms
• language-specific.md: Language-specific reduction techniques for Python, JavaScript, Java, C/C++

Tips

• Start with script: Use provided script for automatic reduction
• Iterate strategies: Try different strategies if first doesn't work well
• Preserve context: Don't over-reduce; keep enough context to understand
• Test determinism: Ensure failure is reproducible before reducing
• Use version control: Commit before reduction to easily revert
• Document oracle: Clearly specify what constitutes failure