qhjqhj00 council-mode-eval
Council Mode: Mitigating Hallucination and Bias in LLMs via Multi-Agent Consensus — Shuai Wu et al. (2026) (arXiv:2604.02923, 2026)
What this evaluates
Evaluates a multi-agent consensus framework's ability to mitigate hallucinations and biases in large language models compared to individual frontier models. It probes factual accuracy, truthfulness, informativeness, and consistency across diverse knowledge domains and varying reasoning complexities.
Datasets
- HaluEval — total ?; splits: test (-1)
- TruthfulQA — total ?; splits: test (-1)
- Multi-Domain Reasoning — total 500; splits: test (500)
Metrics
Hallucination Rate(primary) — range: percent- Percentage of generated claims that are factually incorrect. Calculated as (number of incorrect claims / total claims) * 100.
Truthful Score— range: percent- Percentage of answers rated as truthful by the TruthfulQA evaluation framework.
Informative Score— range: percent- Percentage of answers that are both truthful and informative.
Accuracy— range: percent- Percentage of correct answers on the multi-domain reasoning benchmark.
Bias Variance— range: other- Variance ($\sigma^{2}$) of performance scores across six knowledge domains, measuring consistency.
Input / output format
Input: Natural language queries/prompts from benchmark datasets, routed through a triage classifier to heterogeneous expert models.
Output: Synthesized response generated by the multi-agent consensus framework, or individual model outputs for baselines.
Scoring recipe
def compute_hallucination_rate(predictions, gold_labels):
incorrect = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
if not is_factually_correct(pred, gold):
incorrect += 1
total += 1
return (incorrect / total) * 100 if total > 0 else 0.0
Common pitfalls
- Using external leaderboard scores instead of running unified evaluation scripts via official APIs, which breaks fair comparison.
- Ignoring task complexity scaling; accuracy degrades non-linearly with reasoning steps, so reporting only average accuracy masks performance drops on hard prompts.
- Evaluating bias variance without domain stratification; the benchmark explicitly spans six domains to measure consistency.
Evidence (verbatim from paper)
We report: (1) Hallucination Rate (%): the percentage of generated claims that are factually incorrect; (2) Truthful Score (%): the percentage of answers rated as truthful by the TruthfulQA evaluation framework; (3) Informative Score (%): the percentage of answers that are both truthful and informative; (4) Accuracy (%): correctness on the multi-domain reasoning benchmark; and (5) Bias Variance ($\sigma^{2}$): the variance of performance scores across domains, measuring consistency.
Citation
@misc{wu2026councilmode,
title={Council Mode: Mitigating Hallucination and Bias in LLMs via Multi-Agent Consensus},
author={Shuai Wu et al. (2026)},
year={2026},
note={arXiv:2604.02923}
}
- arXiv: 2604.02923