Deliberative Calibration
Purpose
Produce a fully calibrated ranking when the candidate set is small enough (5-15 items) to allow complete or near-complete pairwise comparison. Leverages parametric models (Bradley-Terry, Thurstone) and structured weighting (AHP) to extract maximum information from each comparison.
When to use
- Candidate count N ≤ 15
- Complete comparison matrix is feasible (N(N-1)/2 pairs manageable)
- High precision required — every rank position matters
- Calibrated strength scores needed, not just ordinal ranking
Budget
| Resource |
Allocation |
| Comparisons |
N(N-1)/2 (complete) or ≥ N×log(N) (near-complete) |
| Iterations |
2-4 rounds (initial + consistency repair) |
| Convergence target |
CR < 0.1, rating stability ≥ 95% |
State Ledger
candidates: [] # list of items being ranked
comparison_matrix: {} # pair → {winner, confidence, reasoning}
ratings: {} # candidate → score
method: "" # bradley-terry | thurstone | ahp | borda
iteration: 0
convergence: {stable: false, score: 0.0}
consistency: {cr: null, cycles: []}
Available Tactics
- adaptive-pair-selection — select next pairs by information gain, compare, update, check convergence
- consistency-audit-loop — verify transitivity, repair inconsistencies
Available SOPs
- pair-selector
- comparison-executor
- rating-update
- convergence-check
- cycle-detection
- inconsistency-localization
- ranking-synthesis
Execution Guidance
- Initialize ratings uniformly for all candidates
- Run adaptive-pair-selection tactic until convergence or complete matrix
- Run consistency-audit-loop to verify transitivity
- If CR > 0.1, re-compare flagged pairs and recompute
- Produce final ranking via ranking-synthesis
Output Format
ranking:
- {rank: 1, candidate: "...", score: 0.95, ci: [0.91, 0.99]}
- {rank: 2, candidate: "...", score: 0.82, ci: [0.77, 0.87]}
method: bradley-terry
consistency_ratio: 0.04
total_comparisons: 28
convergence_iterations: 3
Available Tactics
Optional, no fixed order; the final leaf is always a sop.
| Tactic |
When to use |
| adaptive-pair-selection |
Iteratively select maximally informative pairs, execute comparisons, update ratings, and check convergence until ranking stabilizes. |
| consistency-audit-loop |
Detect preference cycles, localize inconsistent judgments, request corrections, and recompute ratings until consistency threshold is met. |
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP |
When to use |
| ranking-synthesis |
Produce the final ranking artifact from converged ratings and consistency report. |
1---2name: deliberative-calibration3description: Strategy for small-N complete pairwise comparison using Bradley-Terry, Thurstone, AHP, and Borda methods to produce calibrated rankings.4---56# Deliberative Calibration78## Purpose910Produce a fully calibrated ranking when the candidate set is small enough (5-15 items) to allow complete or near-complete pairwise comparison. Leverages parametric models (Bradley-Terry, Thurstone) and structured weighting (AHP) to extract maximum information from each comparison.1112## When to use1314- Candidate count N ≤ 1515- Complete comparison matrix is feasible (N(N-1)/2 pairs manageable)16- High precision required — every rank position matters17- Calibrated strength scores needed, not just ordinal ranking1819## Budget2021| Resource | Allocation |22|----------|-----------|23| Comparisons | N(N-1)/2 (complete) or ≥ N×log(N) (near-complete) |24| Iterations | 2-4 rounds (initial + consistency repair) |25| Convergence target | CR < 0.1, rating stability ≥ 95% |2627## State Ledger2829```yaml30candidates: [] # list of items being ranked31comparison_matrix: {} # pair → {winner, confidence, reasoning}32ratings: {} # candidate → score33method: "" # bradley-terry | thurstone | ahp | borda34iteration: 035convergence: {stable: false, score: 0.0}36consistency: {cr: null, cycles: []}37```3839## Available Tactics4041- **adaptive-pair-selection** — select next pairs by information gain, compare, update, check convergence42- **consistency-audit-loop** — verify transitivity, repair inconsistencies4344## Available SOPs4546- pair-selector47- comparison-executor48- rating-update49- convergence-check50- cycle-detection51- inconsistency-localization52- ranking-synthesis5354## Execution Guidance55561. Initialize ratings uniformly for all candidates572. Run adaptive-pair-selection tactic until convergence or complete matrix583. Run consistency-audit-loop to verify transitivity594. If CR > 0.1, re-compare flagged pairs and recompute605. Produce final ranking via ranking-synthesis6162## Output Format6364```yaml65ranking:66 - {rank: 1, candidate: "...", score: 0.95, ci: [0.91, 0.99]}67 - {rank: 2, candidate: "...", score: 0.82, ci: [0.77, 0.87]}68method: bradley-terry69consistency_ratio: 0.0470total_comparisons: 2871convergence_iterations: 372```7374<!-- BEGIN available-tables (generated) -->7576## Available Tactics7778Optional, no fixed order; the final leaf is always a sop.7980| Tactic | When to use |81| --- | --- |82| adaptive-pair-selection | Iteratively select maximally informative pairs, execute comparisons, update ratings, and check convergence until ranking stabilizes. |83| consistency-audit-loop | Detect preference cycles, localize inconsistent judgments, request corrections, and recompute ratings until consistency threshold is met. |8485## Available SOPs8687Optional, no fixed order; the final leaf is always a sop.8889| SOP | When to use |90| --- | --- |91| ranking-synthesis | Produce the final ranking artifact from converged ratings and consistency report. |9293<!-- END available-tables (generated) -->
yogsoth-ai