Deepimagine Clinical Trial Eval

deepimagine-clinical-trial-eval

DeepImagine: Learning Biomedical Reasoning via Successive Counterfactual Imagining — Zheng et al. (2026) (arXiv:2604.23054, 2026)

What this evaluates

Evaluates a model's ability to perform counterfactual reasoning in biomedical settings by predicting clinical trial outcomes under perturbations of either the outcome measure or the study arm, using similarity-based retrieval to construct counterfactual pairs.

Datasets

• CT open evaluation sample — total 500; splits: test (500)

Metrics

• clinical trial outcome prediction (primary) — range: [0, 1]
  • Accuracy or error rate in predicting the target clinical trial outcome result given a counterfactual perturbation (exact formula not specified in the provided section).

Input / output format

Input: For outcome-measure perturbation: reported result of the most similar alternative outcome measure within the same trial. For study-arm perturbation: reported result of a different study arm measured under the same outcome measure within the same trial.

Output: Predicted result for the target outcome measure or fixed study arm.

Scoring recipe

def score(predictions, gold):
    # Exact metric not specified; assumes classification/regression accuracy
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Evaluation data is sampled from the training split of the source dataset (CT open), risking data leakage if the base model was pre-trained or fine-tuned on it.
• Counterfactual pairs are constructed via embedding similarity (NV-embed), which may introduce retrieval bias or semantic drift not controlled for.
• Two distinct perturbation types (outcome-measure vs. study-arm) are evaluated on the same 500-question pool but reported separately, making aggregate comparison difficult.

Evidence (verbatim from paper)

We evaluate our approach by its downstream utility for clinical trial outcome prediction. From the training split of [[28]], we draw 250 random questions from each of the Superiority and Comparative Effect classes... A evaluated model is provided with the reported result of the nearest outcome measure and asked to predict the target outcome measure’s result under the same study arm.

Citation

@misc{zheng2026deepimagine,
  title={DeepImagine: Learning Biomedical Reasoning via Successive Counterfactual Imagining},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2604.23054}
}

• arXiv: 2604.23054