Benchmarking Skill
Purpose
Measure performance objectively and reproducibly.
Benchmark Design
- Define metric: Latency? Throughput? Accuracy? Cost?
- Choose workload: Representative, diverse, stressful
- Control variables: Same hardware, same data, same config
- Run multiple times: Report mean + stddev / percentiles
- Document everything: Hardware, software, versions, methodology
Metrics
| Metric | Tool | Target |
|---|---|---|
| Latency (P50/P95/P99) | wrk, k6, custom | Per SLO |
| Throughput | wrk, locust | Per SLO |
| GPU util | DCGM, nvidia-smi | > 70% |
| Memory | psutil, py-spy | < 80% |
| Accuracy | Eval harness | Per task |
CF Benchmarks
- Pipeline end-to-end latency per kind
- Model inference latency per model
- Cost per 1K requests per pipeline
- Worker throughput (tasks/hour)
- GPU utilization per node
Process
- Baseline: Measure current state
- Change: Implement optimization
- Measure: Run benchmark again
- Compare: Statistical significance test
- Document: Results in PR / ADR
Quality Gates
- Warmup before measurement
- Multiple runs for variance
- Representative workload
- Documented hardware/config
- Statistical significance (p < 0.05)