Mimic Dos Eval

mimic-dos-eval

CARE: Privacy-Compliant Agentic Reasoning with Evidence Discordance — Haochen Liu et al. (arXiv:2604.01113, 2026)

What this evaluates

Evaluates clinical decision-making under conflicting subjective and objective evidence by testing whether agentic reasoning workflows can correctly classify ICU patient states without collapsing to one-sided predictions.

Datasets

• MIMIC-DOS — total ?; splits: test (-1)

Metrics

• Matthews Correlation Coefficient (MCC) (primary) — range: [-1, 1]
  • Standard correlation coefficient between observed and predicted binary classifications, robust to class imbalance and prediction collapse. Computed as (TPTN - FPFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
• Balanced Accuracy (BA) — range: [0, 1]
  • Average of recall obtained on each class. Computed as (TPR + TNR) / 2.
• G-mean — range: [0, 1]
  • Geometric mean of True Positive Rate and True Negative Rate: sqrt(TPR * TNR).
• Tokens/Sample — range: other
  • Average total token usage (input + output) per evaluated case.

Input / output format

Input: A feature block containing available clinical symptoms, signs, and subjective/objective evidence for an ICU patient case.

Output: A final prediction (binary class/state), optionally accompanied by reasoning steps and confidence scores depending on the specific workflow.

Scoring recipe

def compute_metrics(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == 1 and g == 1)
    tn = sum(1 for p, g in zip(preds, golds) if p == 0 and g == 0)
    fp = sum(1 for p, g in zip(preds, golds) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, golds) if p == 0 and g == 1)
    tpr = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    tnr = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    denom = ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))**0.5
    mcc = (tp*tn - fp*fn) / denom if denom > 0 else 0.0
    ba = (tpr + tnr) / 2.0
    g_mean = (tpr * tnr)**0.5
    return {'MCC': mcc, 'BA': ba, 'G-mean': g_mean, 'TPR': tpr, 'TNR': tnr}

Common pitfalls

• Invalid outputs that do not follow the required format must be excluded from predictive evaluation before calculating metrics.
• Standard accuracy is misleading on this benchmark due to subjective-objective discordance; one-sided prediction collapse must be detected and penalized via metrics like MCC or G-mean.
• Efficiency (Tokens/Sample) must be calculated only over valid, successfully completed runs, not failed or truncated generations.

Evidence (verbatim from paper)

For predictive performance, we report True Positive Rate (TPR), True Negative Rate (TNR), Balanced Accuracy (BA), G-mean, and Matthews Correlation Coefficient (MCC). These metrics are chosen because MIMIC-DOS is a difficult benchmark with subjective–objective discordance, where one-sided prediction collapse can be misleading.

Citation

@misc{liu2026care,
  title={CARE: Privacy-Compliant Agentic Reasoning with Evidence Discordance},
  author={Haochen Liu et al.},
  year={2026},
  note={arXiv:2604.01113}
}

• arXiv: 2604.01113