Compound AI Hw Sw Bench Eval

compound-ai-hw-sw-bench-eval

Benchmarking Compound AI Applications for Hardware-Software Co-Design — Samuthrsindh et al. (2026) (arXiv:2604.09593, 2026)

What this evaluates

This benchmark evaluates hardware-software co-design trade-offs for compound AI applications by measuring end-to-end latency, energy consumption, and accuracy across multi-modal workflows like video QA, evolutionary code generation, and RAG. It probes how different hardware configurations and software optimizations impact system performance under varying latency targets and workload patterns.

Datasets

• Google FRAMES benchmark — total 824; splits: test (824)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match or semantic similarity score comparing the generated answer to the ground truth reference.
• latency — range: seconds
  • Total time from query submission to final response generation, measured in seconds or milliseconds.
• energy consumption — range: joules
  • Total power draw integrated over the execution time of the workflow, measured in joules.

Input / output format

Input: User queries (text, video frames, audio) submitted to the compound AI pipeline, which processes them through modality-specific encoders, LLMs, and vector databases.

Output: Final generated answers or optimized code, along with system-level telemetry metrics (latency, energy, CPU/GPU utilization, cache hit rates) collected via vLLM, SAR, and DCGMI.

Scoring recipe

def evaluate_workflow(workflow, queries, hardware_config):
    total_latency = 0
    total_energy = 0
    correct_answers = 0
    for q in queries:
        start_time = time.now()
        start_power = measure_power()
        response = workflow.run(q, hardware_config)
        end_time = time.now()
        end_power = measure_power()
        latency = end_time - start_time
        energy = integrate_power(start_power, end_power, latency)
        total_latency += latency
        total_energy += energy
        if check_accuracy(response, q.gold):
            correct_answers += 1
    return {
        'avg_latency': total_latency / len(queries),
        'total_energy': total_energy,
        'accuracy': correct_answers / len(queries)
    }

Common pitfalls

• System-level metrics like latency and energy are highly sensitive to background processes and hardware state; the paper notes the need for controlled load generation (Poisson distribution) to simulate realistic usage.
• Accuracy evaluation depends heavily on the embedding model and chunking strategy (e.g., 2,000-token chunks with 200-token overlap), which can significantly alter retrieval quality and downstream LLM performance.

Evidence (verbatim from paper)

For the accuracy–latency tradeoff experiment, documents were embedded using a Text-Embedding-3-Small and split into 2,000-token chunks with a 200-token overlap. The system retrieved the top $k$ chunks, with $k$ ranging from 5 to 30, and was evaluated on the Google FRAMES benchmark across three database configurations: a baseline, a 100-question, and the full 824-question database.

Citation

@misc{samuthrsindh2026compoundai,
  title={Benchmarking Compound AI Applications for Hardware-Software Co-Design},
  author={Samuthrsindh et al. (2026)},
  year={2026},
  note={arXiv:2604.09593}
}

• arXiv: 2604.09593