Claude Agent Benchmark System - Overview

Claude Agent Benchmark System - Overview

What Is This?

A complete benchmarking infrastructure for evaluating Claude agents with:

• ✅ Automated test discovery and execution
• ✅ Comprehensive metrics (Accuracy, F1, Precision, Recall, etc.)
• ✅ Confusion matrix analysis (TP, TN, FP, FN)
• ✅ Timestamped results with full traceability
• ✅ Trend analysis and comparison tools
• ✅ CSV export for external analysis

Quick Start

# 1. Navigate to benchmarks
cd /Users/alexander/dev/benchmarks

# 2. Create a test suite
mkdir my_tests
echo '[{"name":"Test 1","prompt":"What is 2+2?","expected":"4","evaluation_type":"contains"}]' > my_tests/test_basic.json

# 3. Run benchmarks
python run_benchmarks.py

# 4. Analyze results
python analyze_results.py summary

File Structure

benchmarks/
├── run_benchmarks.py          # Main benchmark runner
├── analyze_results.py         # Results analysis & visualization
├── README.md                  # Detailed documentation
├── QUICKSTART.md             # Quick start guide
├── OVERVIEW.md               # This file
├── test_format_example.json  # Example test format
├── .gitignore               # Git ignore patterns
├── results/                 # Auto-generated results (gitignored)
│   └── benchmark_results_YYYY-MM-DD_HH-MM-SS.json
└── [your_test_suites]/      # Your test directories
    └── test_*.json          # Your test files

Core Concepts

1. Test Suites

Directories containing test files. Each suite is benchmarked independently.

2. Test Cases

JSON files containing test definitions with:

• name: Test description
• prompt: Input for Claude
• expected: Expected output
• evaluation_type: How to compare outputs
• context: Optional test configuration

3. Evaluation Types

• exact: String must match exactly
• contains: Expected string must be in output
• semantic: 80%+ word overlap required

4. Metrics Calculated

Classification Metrics

• Accuracy: Overall correctness rate
• Precision: Positive predictive value
• Recall: Sensitivity / True positive rate
• F1 Score: Harmonic mean of precision & recall

Confusion Matrix

• True Positives (TP): Correct positive predictions
• True Negatives (TN): Correct negative predictions
• False Positives (FP): Incorrect positive predictions
• False Negatives (FN): Incorrect negative predictions

Performance Metrics

• Total/passed/failed test counts
• Average execution time per test
• Total execution time

Usage Patterns

Pattern 1: Continuous Testing

# Run after every change
while true; do
  python run_benchmarks.py
  python analyze_results.py compare
  sleep 300
done

Pattern 2: Feature Development

# Create feature-specific tests
mkdir feature_auth_tests
vim feature_auth_tests/test_login.json
python run_benchmarks.py

Pattern 3: Regression Testing

# Run before/after changes
python run_benchmarks.py                    # Before
# ... make changes ...
python run_benchmarks.py                    # After
python analyze_results.py compare           # Compare

Pattern 4: Performance Analysis

# Export and analyze trends
python analyze_results.py trend
python analyze_results.py export

Key Features

1. Automatic Configuration

Loads your Claude settings from ~/.claude.json automatically.

2. Flexible Evaluation

Three evaluation modes support different test types:

• Exact matching for deterministic outputs
• Contains matching for partial content
• Semantic matching for natural language

3. Detailed Outputs

Every benchmark run creates a JSON file with:

• Full metrics breakdown
• Individual test results
• Execution timing
• Configuration snapshot

4. Analysis Tools

Built-in analysis script provides:

• Formatted summaries
• Side-by-side comparisons
• Trend visualization
• CSV export

Best Practices

1. Organize Tests Logically

   • Group related tests into suites
   • Use descriptive directory names
   • Keep test files focused

2. Choose Right Evaluation Type

   • Use exact for deterministic outputs
   • Use contains for key phrases
   • Use semantic for natural language

3. Iterate and Refine

   • Start with simple tests
   • Add complexity gradually
   • Review failed tests carefully

4. Track Over Time

   • Keep all results files
   • Run benchmarks regularly
   • Use trend analysis

5. Document Tests

   • Use descriptive test names
   • Add comments in test files
   • Update README for custom suites

Configuration from .claude.json

The system automatically reads:

• User ID and preferences
• Model configurations
• API settings
• Project-specific settings

Extending the System

Add New Evaluation Types

Edit run_benchmarks.py:

def evaluate_output(self, expected, actual, evaluation_type):
    # Add your custom evaluation logic
    if evaluation_type == 'my_custom_type':
        return your_evaluation_logic()

Add New Metrics

Edit BenchmarkMetrics dataclass:

@dataclass
class BenchmarkMetrics:
    # Add new metrics
    my_custom_metric: float

Then update calculate_metrics() method.

Custom Analysis

Use analyze_results.py as a starting point or load JSON directly:

import json
with open('results/benchmark_results_XXX.json') as f:
    data = json.load(f)
# Perform custom analysis

Output Format

Results are saved as structured JSON:

{
  "timestamp": "2026-02-02T12:00:00",
  "claude_config": {
    "config_loaded": true,
    "user_id": "..."
  },
  "benchmarks": [
    {
      "test_suite": "suite_name",
      "true_positives": 18,
      "true_negatives": 0,
      "false_positives": 0,
      "false_negatives": 2,
      "accuracy": 0.9,
      "precision": 1.0,
      "recall": 0.9,
      "f1_score": 0.947,
      "total_tests": 20,
      "passed_tests": 18,
      "failed_tests": 2,
      "avg_execution_time": 2.34,
      "total_execution_time": 46.8,
      "timestamp": "2026-02-02T12:00:00"
    }
  ]
}

Troubleshooting

Issue: No test suites found

Solution: Ensure test files are named test_*.json or *_test.json

Issue: Claude command not found

Solution: Install Claude CLI and ensure it's in PATH

Issue: All tests timing out

Solution: Check Claude is running and responsive

Issue: JSON parse errors

Solution: Validate JSON with python -m json.tool your_file.json

Next Steps

1. Read QUICKSTART.md for immediate usage
2. Review README.md for detailed documentation
3. Check test_format_example.json for test format
4. Create your first test suite
5. Run benchmarks and iterate

Contributing

This is a flexible framework designed to grow with your needs:

• Add new evaluation methods
• Extend metrics calculation
• Enhance analysis tools
• Add visualization features

Support

• Check documentation in README.md
• Review examples in test_format_example.json
• Read script comments for implementation details
• Modify and extend as needed for your use case

Ready to benchmark your Claude agents?

python run_benchmarks.py