Clinconsensus Eval

clinconsensus-eval

ClinConsensus: A Consensus-Based Benchmark for Evaluating Chinese Medical LLMs across Difficulty Levels — Zheng et al. (2026) (arXiv:2603.02097, 2026)

What this evaluates

Evaluates Chinese medical LLMs on their ability to generate clinically usable, consistent, and safe responses across diverse specialties and difficulty levels. It probes reasoning depth, evidence integration, and longitudinal follow-up rather than raw factual accuracy.

Datasets

• ClinConsensus — total 2500; splits: test (2500)

Metrics

• CACS@7 (primary) — range: percent
  • CACS@$k$ = (100 / (|D| * (N - k + 1))) * Σ max(0, s_i - k + 1), where s_i is the number of rubric criteria met out of N=30, and k=7 is the clinical usability threshold. It measures the normalized expected excess of rubric hits above k-1, emphasizing consistency in producing clinically usable responses.

Input / output format

Input: Full conversation context, candidate model response, and a single rubric criterion $r_j$.

Output: JSON object with two fields: 'explanation' (string) and 'criteria_met' (boolean).

Scoring recipe

def compute_cacs(judge_outputs, N=30, k=7):
    total_excess = 0
    for out in judge_outputs:
        s_i = sum(1 for r in out if r['criteria_met'])
        total_excess += max(0, s_i - k + 1)
    cacs = (100 / (len(judge_outputs) * (N - k + 1))) * total_excess
    return cacs

Common pitfalls

• Interpreting k=7 as a strict passing score out of 30, rather than a truncation point for cumulative survival aggregation.
• Relying on average rubric accuracy, which masks clinically unusable responses that consistently score near the pass mark.
• Evaluating completeness against the full 30-item rubric instead of focusing on the high-score region above the usability threshold.

Evidence (verbatim from paper)

We define CACS@$k$ as the truncated area under the reverse-ECDF curve from a clinical threshold $k$ to the maximum score $N$: ... By summing only over $t\geq k$, CACS@$k$ assigns zero marginal credit to responses that do not reach $k$ rubric hits, thereby emphasizing consistency in producing clinically usable responses.

Citation

@misc{zheng2026clinconsensus,
  title={ClinConsensus: A Consensus-Based Benchmark for Evaluating Chinese Medical LLMs across Difficulty Levels},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2603.02097}
}

• arXiv: 2603.02097