Doctor Rec Eval

doctor-rec-eval

Extreme Multilabel Classification for Specialist Doctor Recommendation with Implicit Feedback and Limited Patient Metadata — Valdeira et al. (2023) (arXiv:2308.11022, 2023)

What this evaluates

This evaluation probes a model's ability to recommend specialist doctors for patients using implicit interaction data and limited demographic metadata. It specifically tests performance in both warm-start (seen patients) and cold-start (new patients) scenarios, emphasizing the model's capacity to handle popularity bias and recommend less popular specialists.

Datasets

• Doctor Recommendation Dataset — total 2890042; splits: train (-1), test_seen (-1), test_new (-1)

Metrics

• PS-nDCG@3 (primary) — range: [0, 1]
  • Propensity-scored Normalized Discounted Cumulative Gain at rank 3. It weights the relevance of recommended doctors by inverse propensity scores to downweight popular doctors and emphasize tail labels, normalized by the ideal DCG.

Input / output format

Input: Patient demographic and interaction history features (encoded as TF-IDF-like vectors), doctor attributes, and a patient-doctor interaction matrix. For baselines, the rating matrix R is provided; for the proposed XML model, patient and doctor feature matrices are concatenated.

Output: A ranked list of top-K specialist doctors (K=30 for the XML model; full ranked list for baselines).

Scoring recipe

def compute_ps_ndcg_at_k(predictions, gold, k=3, propensities=None):
    dcg = 0.0
    for i, pred in enumerate(predictions[:k]):
        if pred in gold:
            rel = 1.0
            if propensities:
                rel *= propensities.get(pred, 1.0)
            dcg += rel / math.log2(i + 2)
    ideal_rels = sorted([propensities.get(g, 1.0) for g in gold if g in predictions[:k]], reverse=True)
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal_rels))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Ignoring propensity scoring leads to severe popularity bias, as models will only recommend already well-known doctors with shorter wait times.
• XML models only output top-K predictions (K=30); comparing them fairly requires filtering out patients who do not receive at least 3 predictions per specialization.
• Cold-start (new patient) evaluation requires a specific feature scenario (S4) that excludes hospital features unavailable for unseen users, unlike the warm-start scenario (S5).

Evidence (verbatim from paper)

We employ standard ranking metrics for the evaluation of RS and XML, namely Normalized Discounted Cumulative Gain@K (nDCG@K) Yan et al. ([2020]), Precision@K (P@K) Peito and Han ([2021]), and Recall@K Deng and Huangfu ([2019]). Additionally, we incorporate their propensity-scored counterparts of the first two (PSnDCG@K and PSP@K, respectively), which account for label popularity Bhatia et al. ([2016]). That is, they prevent a good performance at the cost of predicting head labels and put emphasis on tail labels. In our setting, this is crucial as popular doctors are often already well-known to patients and may have longer waiting times. Thus, accurately recommending less popular doctors becomes highly relevant. Here, we represent PSnDCG@3 and Recall@10, but the remaining metrics lead to similar conclusions and can be found in the Supplementary Material.

Citation

@misc{valdeira2023extreme,
  title={Extreme Multilabel Classification for Specialist Doctor Recommendation with Implicit Feedback and Limited Patient Metadata},
  author={Valdeira et al. (2023)},
  year={2023},
  note={arXiv:2308.11022}
}

• arXiv: 2308.11022