Test Generation Benchmark Analysis

Test Generation Benchmark Analysis

Executive Summary

This benchmark compared two approaches to generating unit tests using PDD:

1. Code-based (Original): Tests generated by analyzing actual implementation code
2. Example-based (TDD): Tests generated from usage examples showing intended behavior

Key Finding: The TDD/example-based approach generated more comprehensive, portable tests (37 vs 20 test functions) while maintaining the same focus on public API testing.

Benchmark Setup

Module Under Test: Email Validator

A simple email validator with:

• Public API: EmailValidator.validate_email(email: str) -> ValidationResult
• Internal helpers (intentionally added to test if code-based approach references them):
  • _normalize() - strips whitespace, lowercases
  • _check_local_part() - validates part before @
  • _check_domain() - validates part after @
  • _has_consecutive_dots() - checks for .. pattern
  • EMAIL_PATTERN - compiled regex class attribute

File Generation

Test Generation Commands

# Code-based: Analyzes actual implementation
pdd test email_validator_python.prompt src/email_validator.py

# Example-based (TDD): Analyzes usage examples
pdd test email_validator_python.prompt examples/email_validator_example.py

Quantitative Results

Test Statistics

Implementation Detail References

Both approaches were tested for references to internal implementation details:

grep -E '\._normalize|\._check_local_part|\._check_domain|\._has_consecutive_dots|\.EMAIL_PATTERN'

Finding: Neither approach referenced private implementation details. Both prompts include instructions to avoid testing internal implementation, and both followed this guidance.

Qualitative Analysis

Test Structure Comparison

Code-based Approach

class TestEmailValidatorUnit:
    @pytest.fixture
    def validator(self):
        return EmailValidator()

    def test_valid_email_basic(self, validator):
        """Test basic valid email addresses."""
        test_cases = [
            "user@example.com",
            "user.name@example.com",
            "user+tag@example.org",
        ]
        for email in test_cases:
            result = validator.validate_email(email)
            assert result.is_valid

Characteristics:

• Class-based organization
• Batched test cases in loops
• Multiple assertions per test function
• Fewer, broader test functions

Example-based Approach

def test_valid_standard_email(validator):
    """Test a standard valid email address."""
    email = "user@example.com"
    result = validator.validate_email(email)

    assert result.is_valid is True
    assert result.normalized_email == "user@example.com"
    assert result.error_message is None

def test_valid_email_with_plus_tag(validator):
    """Test valid email with plus tag in local part."""
    email = "user+tag@example.org"
    result = validator.validate_email(email)

    assert result.is_valid is True

Characteristics:

• Flat function structure
• One test case per function
• Self-contained tests
• More granular test coverage

Portability Assessment

Test Coverage Comparison

Validation Rules Coverage

Additional Edge Cases

Z3 Formal Verification

Both approaches attempted to generate Z3 formal verification tests. Both failed with the same API errors:

# Generated code (incorrect):
email.contains("@")  # AttributeError: 'SeqRef' has no attribute 'contains'
email.index("@")     # AttributeError: 'SeqRef' has no attribute 'index'

# Correct Z3 API:
Contains(email, "@")
IndexOf(email, "@")

Finding: This is a model hallucination issue affecting both approaches equally. The prompts request Z3 tests, but the model generates incorrect Z3 API calls.

When to Use Each Approach

Use TDD/Example-based When:

1. Writing tests first (true TDD)

   • You have a prompt describing intent
   • You have an example showing usage
   • Implementation doesn't exist yet

2. API contract is the priority

   • Tests should verify public interface
   • Implementation details may change
   • Multiple implementations possible

3. Iterative development

   • Rapid prototyping
   • Frequent refactoring expected
   • Design is evolving

4. Maximum test coverage desired

   • Generated 85% more test functions
   • More granular failure detection

Use Code-based When:

1. Testing existing code

   • No example file exists
   • Need to verify what code actually does
   • Legacy system documentation

2. Coverage-driven testing

   • Want tests that cover code paths
   • Need to identify dead code
   • Measuring implementation coverage

3. Verifying specific behaviors

   • Bug reproduction
   • Regression testing
   • Implementation-specific edge cases

Recommendations

For New Development

Use the TDD/example-based approach:

# 1. Write prompt describing intent
vim my_module_python.prompt

# 2. Generate example file showing usage
pdd example my_module_python.prompt src/my_module.py --output examples/my_module_example.py
# Or write example manually if implementation doesn't exist

# 3. Generate tests from example
pdd test my_module_python.prompt examples/my_module_example.py

# 4. Generate/iterate on implementation
pdd generate my_module_python.prompt

# 5. Run tests until passing
pytest tests/test_my_module.py

For Existing Code

Use the code-based approach:

# Generate tests directly from implementation
pdd test my_module_python.prompt src/my_module.py

Improving Test Quality

1. Fix Z3 tests manually or remove them if formal verification not needed

2. Consider relaxing error message assertions for portability:

   # Brittle:
   assert result.error_message == "Email must contain exactly one @ symbol"
   
   # More portable:
   assert result.error_message is not None
   assert "@" in result.error_message.lower()
   

3. Add the example file to version control - it serves as executable documentation

Conclusion

The TDD/example-based approach produces more comprehensive, portable tests that align better with the PDD philosophy of defining intent rather than testing implementation accidents. While both approaches avoided referencing private implementation details in this benchmark, the example-based approach generated 85% more test functions with better isolation and failure localization.

Bottom line: For new PDD development, prefer the TDD/example-based approach. Reserve code-based testing for legacy code or when you specifically need to verify implementation behavior.

Appendix: Reproducing This Benchmark

cd examples/test_generation_benchmark

# Clean previous results
make clean

# Run full benchmark
make benchmark

# View analysis
make analyze

# Compare generated tests
make compare

# Run test suites
make run-tests

Files in This Benchmark

test_generation_benchmark/
├── email_validator_python.prompt      # Intent specification
├── src/email_validator.py             # Implementation (pdd generate + manual helpers)
├── examples/email_validator_example.py # Usage examples (pdd example)
├── Makefile                           # Automation
├── README.md                          # Instructions
├── ANALYSIS.md                        # This file
└── benchmark_results/
    ├── code_based/test_email_validator.py
    └── example_based/test_email_validator.py