Ic Index

ic-index

Interaction Concordance Index: Performance Evaluation for Interaction Prediction Methods — Pahikkala et al. (2025) (arXiv:2510.14419, 2025)

What this evaluates

Evaluates whether machine learning models can correctly capture non-additive interaction effects in drug-target affinity prediction, rather than merely learning global means or individual drug/target main effects. It measures the proportion of correctly predicted interaction directions across test pairs.

Datasets

• Davis — total 30056; splits: train (-1), val (-1), test (-1); repo https://github.com/TurkuML/IC-index-experiments
• Metz — total 93356; splits: train (-1), val (-1), test (-1); repo https://github.com/TurkuML/IC-index-experiments
• KIBA — total 118254; splits: train (-1), val (-1), test (-1); repo https://github.com/TurkuML/IC-index-experiments
• Merget — total 167995; splits: train (-1), val (-1), test (-1); repo https://github.com/TurkuML/IC-index-experiments
• GPCR — total 21185; splits: train (-1), val (-1), test (-1); repo https://github.com/TurkuML/IC-index-experiments
• Ion Channels — total 42840; splits: train (-1), val (-1), test (-1); repo https://github.com/TurkuML/IC-index-experiments
• Enzymes — total 295480; splits: train (-1), val (-1), test (-1); repo https://github.com/TurkuML/IC-index-experiments

Metrics

• IC-index (primary) — range: [0, 1]
  • Proportion of correctly predicted interaction directions. It evaluates whether the model correctly identifies if the effect of a drug on a target depends on the target (interaction effect), rather than just main effects.

Input / output format

Input: Drug-target pairs represented by feature matrices (similarity/kernels for drugs and targets) and categorical identities. Known DTA values are provided for training.

Output: Predicted drug-target affinity values (continuous) or binary interaction labels, depending on the dataset type.

Scoring recipe

correct_directions = 0
total_pairs = 0
for d, t in test_pairs:
    pred_affinity = model.predict(d, t)
    true_affinity = gold_affinity[d, t]
    if correctly_predicts_interaction_direction(pred_affinity, true_affinity):
        correct_directions += 1
    total_pairs += 1
ic_index = correct_directions / total_pairs
# Average fold-wise values across 9-fold CV

Common pitfalls

• Standard metrics like C-index or accuracy can be inflated by models that only learn grand means or main effects, masking their failure to capture true interactions.
• Randomized algorithms on small test sets may yield falsely promising results due to variance, even when expected performance is random.
• Permutation equivariance in training data prevents models from learning specific drug-target interactions, especially when evaluating on unseen entities.

Evidence (verbatim from paper)

Better than random average IC-index values are obtained only with PS and PR, because they are the only learning algorithms able to infer nonadditive predictors. Moreover, this takes place only on IDIT data, because no side information beyond the categorical drug and target identities is available. ... Overall, the results demonstrate that for all considered performance measures except IC-index, fairly trivial methods can achieve good performance simply by modeling grand mean, drug main or target main effects.

Citation

@misc{pahikkala2025interactionconcordanceindex,
  title={Interaction Concordance Index: Performance Evaluation for Interaction Prediction Methods},
  author={Pahikkala et al. (2025)},
  year={2025},
  note={arXiv:2510.14419}
}

• arXiv: 2510.14419