Pytrial Eval

pytrial-eval

PyTrial: Machine Learning Software and Benchmark for Clinical Trial Applications — Wang et al. (2023) (arXiv:2306.04018, 2023)

What this evaluates

Evaluates machine learning models across clinical trial tasks including patient and trial outcome prediction, trial search, and patient simulation, using standardized tabular and sequential data formats.

Datasets

• Tabular Clinical Trial Patient Datasets — total ?; splits: val (-1), test (-1)
• TOP Benchmark — total ?; splits: val (-1), test (-1)
• Trial Similarity Dataset — total ?; splits: val (-1), test (-1)
• Sequential Trial Patient Data — total ?; splits: val (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve; measures the model's ability to rank positive instances higher than negative ones across all classification thresholds.
• PR-AUC — range: [0, 1]
  • Area under the precision-recall curve; emphasizes performance on the positive class, particularly useful for imbalanced datasets.
• Precision@K — range: [0, 1]
  • Fraction of relevant items among the top K retrieved items.
• Recall@K — range: [0, 1]
  • Fraction of all relevant items that appear in the top K retrieved items.
• nDCG@K — range: [0, 1]
  • Normalized Discounted Cumulative Gain at K; measures ranking quality by weighting relevant items higher when they appear earlier in the list.
• Pearson correlation (r) — range: [-1, 1]
  • Linear correlation coefficient between dimension-wise probabilities of real and synthetic data; used to evaluate fidelity of generated patient records.

Input / output format

Input: Tabular patient records for outcome prediction; tabular trial metadata for trial outcome prediction; trial document text for search; sequential patient event logs for simulation.

Output: Predicted probability scores or binary labels; ranked list of candidate trials; synthetic sequential patient records.

Scoring recipe

def compute_metrics(y_true, y_pred, k=5):
    auroc = roc_auc_score(y_true, y_pred)
    pr_auc = average_precision_score(y_true, y_pred)
    top_k = np.argsort(y_pred)[-k:][::-1]
    prec_k = np.mean([y_true[i] for i in top_k])
    rec_k = prec_k * len(top_k) / max(np.sum(y_true), 1)
    dcg = sum((2**y_true[i] - 1) / np.log2(i + 2) for i in top_k)
    idcg = sum((2**1 - 1) / np.log2(i + 2) for i in range(k))
    ndcg_k = dcg / max(idcg, 1e-9)
    return auroc, pr_auc, prec_k, rec_k, ndcg_k

def compute_fidelity(real_data, synth_data):
    real_probs = np.mean(real_data, axis=0)
    synth_probs = np.mean(synth_data, axis=0)
    return np.corrcoef(real_probs, synth_probs)[0, 1]

Common pitfalls

• Deep learning models may fail to converge on small tabular datasets (e.g., FT-Transformer struggled on two datasets).
• Synthetic data fidelity is evaluated via dimension-wise probability correlation rather than standard generative quality metrics.
• Hyperparameters are selected based on validation performance, which may lead to optimistic test results if the validation set is not held out properly.

Evidence (verbatim from paper)

We evaluate the patient outcome prediction algorithms on the tabular clinical trial patient datasets released in (Wang et al., 2023a)... The result of AUROC is shown in Table 3. The results, including AUROC and PR-AUC scores, are presented in Table 4. Results are precision@K (prec@K) and recall@K (rec@K), and nDCG@K for trial similarities (ranking).

Citation

@misc{wang2023pytrial,
  title={PyTrial: Machine Learning Software and Benchmark for Clinical Trial Applications},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2306.04018}
}

• arXiv: 2306.04018