cascadeflow Benchmarking Guide

cascadeflow Benchmarking Guide

Comprehensive guide to using and creating benchmarks for cascadeflow cascade systems.

Table of Contents

1. Overview
2. Available Benchmarks
3. Running Benchmarks
4. Understanding Results
5. Creating Custom Benchmarks
6. Best Practices

Overview

cascadeflow includes a comprehensive benchmarking suite to evaluate cascade system performance across multiple dimensions:

• Code Generation (HumanEval)
• Math Reasoning (GSM8K)
• Multi-Turn Conversations (MT-Bench)
• Factual Accuracy (TruthfulQA)
• Customer Support (Real-world ROI)
• Provider Comparison (OpenAI vs Anthropic)

Why Benchmark?

Benchmarks help you:

1. Validate Quality: Ensure cascade maintains accuracy
2. Measure Cost Savings: Quantify actual ROI
3. Tune Thresholds: Find optimal quality thresholds
4. Compare Providers: Understand provider-specific behavior
5. Track Performance: Monitor improvements over time

Available Benchmarks

1. HumanEval: Code Generation

Purpose: Tests code generation capabilities

Dataset: 10 programming problems from HumanEval

Metrics:

• Code correctness (passes unit tests)
• Acceptance rate (drafter usage)
• Cost reduction vs always using powerful model

Usage:

cd tests/benchmarks
python -m benchmarks.humaneval

Expected Results:

• Accuracy: 70-85%
• Acceptance Rate: 50-70%
• Cost Reduction: 40-60%

2. GSM8K: Math Reasoning

Purpose: Evaluates multi-step mathematical reasoning

Dataset: 10 grade school math word problems

Metrics:

• Answer correctness (numerical accuracy)
• Drafter accuracy on accepted problems
• Cost savings on math tasks

Usage:

python -m benchmarks.gsm8k

Expected Results:

• Accuracy: 75-90%
• Acceptance Rate: 40-60%
• Cost Reduction: 35-55%

3. MT-Bench: Multi-Turn Conversations

Purpose: Tests context retention and dialogue coherence

Dataset: 10 multi-turn conversations across 8 categories

Categories: Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, Humanities

Metrics:

• All-turns-pass rate (conversation success)
• Per-turn quality scores
• Context retention across turns

Usage:

python -m benchmarks.mtbench

Expected Results:

• Accuracy: 70-80% (all turns pass)
• Acceptance Rate: 50-70%
• Cost Reduction: 40-60%

4. TruthfulQA: Factual Accuracy

Purpose: Evaluates truthfulness and avoids misinformation

Dataset: 15 questions across 10 categories (misconceptions, science, history, health, etc.)

Metrics:

• Truthful answer rate
• Misinformation detection
• Verifier escalation effectiveness

Usage:

python -m benchmarks.truthfulqa

Expected Results:

• Accuracy: 70-85% truthful
• Acceptance Rate: 55-70%
• Cost Reduction: 40-60%

5. Customer Support: Real-World ROI

Purpose: Demonstrates practical business value

Dataset: 20 realistic support scenarios

Distribution:

• 60% Simple FAQs
• 20% Moderate complexity
• 20% Complex escalations

Metrics:

• Response helpfulness
• Professional tone
• Monthly/annual ROI projections

Usage:

python -m benchmarks.customer_support

Expected Results:

• Accuracy: 85%+ helpful
• Acceptance Rate: 60-70%
• Annual Savings: $5K-$15K (at 10K queries/month)

6. Provider Comparison

Purpose: Compares quality scoring across providers

Tests: OpenAI vs Anthropic verifiers with same drafter

Metrics:

• Quality score agreement
• Latency differences
• Cost differences
• Optimal thresholds per provider

Usage:

python -m benchmarks.provider_comparison

Key Findings:

• Agreement rate: 70-85%
• Score variance: ±0.1-0.15
• Provider-specific thresholds may be needed

Running Benchmarks

Single Benchmark

Run individual benchmarks:

cd tests/benchmarks

# HumanEval
python -m benchmarks.humaneval

# GSM8K
python -m benchmarks.gsm8k

# MT-Bench
python -m benchmarks.mtbench

# TruthfulQA
python -m benchmarks.truthfulqa

# Customer Support
python -m benchmarks.customer_support

# Provider Comparison
python -m benchmarks.provider_comparison

All Benchmarks

Run the complete suite:

python -m benchmarks.run_all --output-dir results --format md,json,csv

Output:

• results/comparison.md - Comparison table
• results/results.json - Raw data
• results/*.csv - Per-benchmark CSVs (if implemented)

With Custom Configuration

from benchmarks.humaneval import HumanEvalBenchmark

benchmark = HumanEvalBenchmark(
    drafter_model="gpt-4o-mini",
    verifier_model="gpt-4o",
    quality_threshold=0.75,  # Custom threshold
    max_samples=20,           # More samples
)

summary = await benchmark.run()

Understanding Results

Key Metrics

Accuracy

Percentage of correct/helpful responses

• High (>85%): Production-ready quality
• Medium (70-85%): Consider raising threshold
• Low (<70%): Quality issues, investigate further

Acceptance Rate

Percentage of queries handled by drafter

• High (>70%): Excellent cost savings
• Medium (50-70%): Good balance
• Low (<50%): Threshold may be too strict

Cost Reduction

Percentage savings vs always using verifier

• Calculation: (baseline_cost - cascade_cost) / baseline_cost * 100
• Target: 40-65% for most use cases

Drafter Accuracy

Accuracy when drafter is accepted

• High (>80%): Drafter reliable on simple tasks
• Low (<70%): Quality threshold may be too lenient

Sample Output

==================== CUSTOMER SUPPORT BENCHMARK RESULTS ====================

Total Queries:       20
Helpful Responses:   17 (85.0%)
Drafter Accepted:    12 (60.0%)
Verifier Escalated:  8 (40.0%)

Cost Analysis:
  Cascade Total Cost:  $0.003450
  Baseline Total Cost: $0.008900
  Cost Savings:        $0.005450 (61.2%)

  💰 Projected ROI (10K queries/month):
     Monthly Savings:  $272.50
     Annual Savings:   $3,270.00

Performance:
  Average Latency:     1250ms
  Average Quality:     0.823
  Drafter Accuracy:    91.7% (when accepted)

Key Findings:
  ✅ Drafter handles 60% of queries (meets 60% target)
  ✅ High response quality: 85% helpful
  💰 Significant cost savings: 61% reduction
  ✅ Drafter maintains quality: 92% accurate on simple queries

📊 Business Impact:
  - Cascade pattern saves $3,270/year at 10K queries/month
  - Drafter handles 60% of queries at 10x lower cost
  - Verifier escalation ensures complex queries get premium responses
  - ✅ Ready for production deployment in customer support use cases

Interpreting Trade-offs

High Accuracy, Low Acceptance

• Quality threshold too strict
• Missing cost savings opportunity
• Action: Lower threshold slightly (e.g., 0.7 → 0.65)

High Acceptance, Low Accuracy

• Quality threshold too lenient
• Drafter producing low-quality responses
• Action: Raise threshold (e.g., 0.7 → 0.75)

Optimal Balance

• Accuracy: 80-90%
• Acceptance: 50-70%
• Cost Reduction: 45-65%

Creating Custom Benchmarks

Step 1: Extend Base Class

from benchmarks.base import Benchmark, BenchmarkResult, BenchmarkSummary
from typing import Any

class MyBenchmark(Benchmark):
    def __init__(
        self,
        drafter_model: str = "gpt-4o-mini",
        verifier_model: str = "gpt-4o",
        quality_threshold: float = 0.7,
        max_samples: int = 10,
    ):
        super().__init__(
            dataset_name="MyBenchmark-10",
            drafter_model=drafter_model,
            verifier_model=verifier_model,
            baseline_model=verifier_model,
            quality_threshold=quality_threshold,
            max_samples=max_samples,
        )

Step 2: Load Dataset

def load_dataset(self) -> list[tuple[str, Any]]:
    """Load your benchmark data."""

    problems = [
        {
            "problem_id": "BENCH/1",
            "query": "Your test query here",
            "expected_answer": "The correct answer",
            "category": "category_name",
        },
        # ... more problems
    ]

    return [(p["problem_id"], p) for p in problems[:self.max_samples]]

Step 3: Implement Evaluation

def evaluate_prediction(
    self, prediction: str, ground_truth: Any
) -> tuple[bool, float]:
    """
    Evaluate if prediction is correct.

    Args:
        prediction: Model's response
        ground_truth: Problem data with expected answer

    Returns:
        (is_correct, quality_score)
    """
    # Your evaluation logic here
    expected = ground_truth["expected_answer"]
    is_correct = expected.lower() in prediction.lower()

    # Calculate quality score (0.0 to 1.0)
    quality_score = 1.0 if is_correct else 0.0

    return is_correct, quality_score

Step 4: Run Cascade (Optional Override)

async def run_cascade(self, query: str) -> dict[str, Any]:
    """Run cascade on a query (optional override)."""

    from cascadeflow.agent import CascadeAgent

    agent = CascadeAgent(
        models=[
            {"name": self.drafter_model, "provider": "openai"},
            {"name": self.verifier_model, "provider": "openai"},
        ],
        quality={"threshold": self.quality_threshold},
    )

    result = await agent.arun(query)
    return result

Step 5: Create Runner Function

async def run_my_benchmark() -> BenchmarkSummary:
    """Run benchmark and print results."""

    print("\\n" + "=" * 80)
    print("MY CUSTOM BENCHMARK")
    print("=" * 80 + "\\n")

    benchmark = MyBenchmark(
        drafter_model="gpt-4o-mini",
        verifier_model="gpt-4o",
        quality_threshold=0.7,
        max_samples=10,
    )

    summary = await benchmark.run()

    # Print results
    print(f"Accuracy: {summary.accuracy*100:.1f}%")
    print(f"Cost Savings: ${summary.cost_savings:.6f}")

    return summary


if __name__ == "__main__":
    import asyncio
    asyncio.run(run_my_benchmark())

Best Practices

1. API Key Management

Always set API keys before running benchmarks:

export OPENAI_API_KEY="your-key-here"
export ANTHROPIC_API_KEY="your-key-here"  # For provider comparison

2. Start Small

Test with max_samples=5 before running full benchmarks:

benchmark = MyBenchmark(max_samples=5)  # Quick test

3. Monitor Costs

Track costs during benchmark runs:

print(f"Total Cost: ${summary.total_cost:.6f}")
print(f"Cost per Query: ${summary.total_cost / summary.total_tests:.6f}")

4. Iterate on Thresholds

Test multiple thresholds to find optimal:

for threshold in [0.6, 0.65, 0.7, 0.75, 0.8]:
    benchmark = MyBenchmark(quality_threshold=threshold)
    summary = await benchmark.run()
    print(f"Threshold {threshold}: Accuracy={summary.accuracy:.2f}, "
          f"Acceptance={summary.acceptance_rate:.2f}")

5. Compare Providers

Test both OpenAI and Anthropic to find best fit:

# OpenAI verifier
openai_bench = MyBenchmark(verifier_model="gpt-4o")

# Anthropic verifier
anthropic_bench = MyBenchmark(verifier_model="claude-sonnet-4-5-20250929")

6. Track Over Time

Save results to compare improvements:

results_history = []

summary = await benchmark.run()
results_history.append({
    "date": datetime.now().isoformat(),
    "accuracy": summary.accuracy,
    "cost_reduction": summary.cost_reduction_pct,
})

7. Category Analysis

Analyze performance by category:

# In your benchmark
category_stats = {}
for result in self.results:
    cat = result.category
    if cat not in category_stats:
        category_stats[cat] = {"correct": 0, "total": 0}
    category_stats[cat]["total"] += 1
    if result.correct:
        category_stats[cat]["correct"] += 1

for cat, stats in category_stats.items():
    accuracy = stats["correct"] / stats["total"]
    print(f"{cat}: {accuracy*100:.1f}% ({stats['correct']}/{stats['total']})")

Troubleshooting

Low Accuracy

Symptoms: Accuracy <70%

Possible Causes:

1. Quality threshold too low
2. Drafter model insufficient for task
3. Evaluation criteria too strict

Solutions:

• Raise quality threshold to 0.75-0.8
• Use stronger drafter (e.g., gpt-4o instead of gpt-4o-mini)
• Review evaluation logic

Low Acceptance Rate

Symptoms: Acceptance <40%

Possible Causes:

1. Quality threshold too high
2. Drafter model overly conservative
3. Domain mismatch

Solutions:

• Lower threshold to 0.6-0.65
• Check drafter model capabilities
• Use domain-specific prompting

High Costs

Symptoms: Cost reduction <30%

Possible Causes:

1. Too many verifier escalations
2. Expensive baseline model
3. Large context windows

Solutions:

• Lower quality threshold
• Use cheaper baseline for comparison
• Optimize prompt length

Inconsistent Results

Symptoms: Results vary significantly between runs

Possible Causes:

1. Model randomness (temperature)
2. Small sample size
3. Edge cases in dataset

Solutions:

• Increase max_samples for stability
• Set temperature=0 for deterministic results
• Review dataset for outliers

Additional Resources

• Source Code: tests/benchmarks/
• Examples: Each benchmark file includes usage examples
• API Documentation: See cascadeflow.agent.CascadeAgent docs

For questions or issues, please open an issue on GitHub.

Happy Benchmarking! 🎯