gabrielmoreira Comparative Evaluation
Absolute quality scores are useful but limited. Comparative evaluation — putting outputs side by side and asking which is better — often reveals quality differences that rubrics miss.
Comparison Methods
- A/B testing: Show different users different versions and compare outcomes
- Side-by-side evaluation: Show evaluators two outputs for the same input and ask which is better
- Preference ranking: Show evaluators multiple outputs and rank them from best to worst
- Paired comparison: Compare every pair of options to build a complete ranking
- Elo rating: Use tournament-style comparisons to develop continuous quality scores
Designing A/B Tests for AI
A/B testing AI is different from A/B testing UI:
- Variance is high: The same prompt can produce different outputs, so you need more samples
- Context matters: The same change might help for one task and hurt for another
- Metrics lag: AI quality changes may take time to show up in user behavior
- Interaction effects: A change to one part of the conversation affects all subsequent parts Design A/B tests with:
- Sufficient sample sizes to account for output variance
- Segmentation by task type and user experience level
- Multiple metrics (don't optimise for one at the expense of others)
- Guardrails to catch severe quality regressions quickly
Side-by-Side Evaluation Design
For human evaluation of AI outputs:
- Blind evaluation: Evaluators shouldn't know which version is which
- Consistent inputs: Compare outputs generated from the same input
- Structured criteria: Give evaluators specific dimensions to compare on, not just "which is better"
- Multiple evaluators: Use at least 3 evaluators per comparison for reliability
- Diverse inputs: Test across a representative sample of real user inputs
When to Use Comparative vs. Absolute Evaluation
- Comparative: Best for choosing between alternatives, detecting subtle quality differences, and model selection
- Absolute: Best for measuring against a standard, tracking progress over time, and certification
Design Artefacts
- A/B test design templates
- Side-by-side evaluation protocols
- Evaluator instructions and rubrics
- Sample size calculators for AI experiments
- Comparison result analysis frameworks