yogsoth-ai Collective Adjudication
Purpose
Aggregate rankings from multiple independent judges into a single consensus ranking. Handles disagreement detection, voting paradoxes, and produces transparent aggregation with disagreement maps.
When to use
- Multiple judges/evaluators available (≥3)
- LLM-as-judge with multiple prompting perspectives
- Committee decision-making requiring formal aggregation
- Need to identify and characterize disagreement patterns
Budget
| Resource | Allocation |
|---|---|
| Judges/Perspectives | ≥3 independent evaluators |
| Comparisons per judge | Complete or near-complete per judge |
| Aggregation methods | ≥2 methods for robustness check |
| Disagreement threshold | Flag pairs where judges disagree >40% |
State Ledger
candidates: []
perspectives: [] # judge identities/prompts
ballots: [] # [{judge, ranking: [...]}]
aggregation_results: {} # method → consensus_ranking
disagreement_map: {} # pair → {agreement_rate, split}
cycles: [] # Condorcet cycles if any
method: "" # schulze | borda | kemeny-young | copeland
Available Tactics
- multi-judge-aggregation — collect ballots, aggregate, identify disagreement
- consistency-audit-loop — detect cycles in aggregated preferences
Available SOPs
- ballot-collection
- aggregation-method
- cycle-detection
- inconsistency-localization
- ranking-synthesis
Execution Guidance
- Define perspectives (judge roles, prompting strategies)
- Run ballot-collection to gather independent rankings
- Run aggregation-method with primary method (Schulze recommended)
- Run cycle-detection on aggregated pairwise matrix
- If cycles exist, run inconsistency-localization
- Cross-validate with secondary method (Borda or Copeland)
- Produce final ranking with disagreement heatmap
Output Format
consensus_ranking:
- {rank: 1, candidate: "...", wins: 8, copeland_score: 0.95}
- {rank: 2, candidate: "...", wins: 7, copeland_score: 0.88}
method: schulze
judges: 5
condorcet_winner: "candidate_a" # or null if cycle
disagreement_hotspots:
- {pair: ["c", "d"], agreement: 0.4, split: "3:2"}
cross_validation: {borda_agreement: 0.92, copeland_agreement: 0.96}
Available Tactics
Optional, no fixed order; the final leaf is always a sop.
| Tactic | When to use |
|---|---|
| consistency-audit-loop | Detect preference cycles, localize inconsistent judgments, request corrections, and recompute ratings until consistency threshold is met. |
| multi-judge-aggregation | Collect independent rankings from multiple judges, aggregate using social choice methods, and identify disagreement hotspots. |
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| ranking-synthesis | Produce the final ranking artifact from converged ratings and consistency report. |