Ce Recommender Benchmark Eval

ce-recommender-benchmark-eval

From Top-1 to Top-K: A Reproducibility Study and Benchmarking of Counterfactual Explanations for Recommender Systems — Nguyen et al. (2026) (arXiv:2604.19663, 2026)

What this evaluates

Evaluates the quality, sparsity, and computational efficiency of counterfactual explanation methods for recommender systems. It probes how effectively explanations can alter recommendation rankings, how interpretable the generated explanations are, and the cost of generating them across different input formats and perturbation scopes.

Datasets

• Recommender system interaction datasets — total ?; splits: test (-1)

Metrics

• POS-P@K (primary) — range: [0, 1]
  • Measures how quickly a target item drops out of the top-K list when interactions are removed in decreasing order of importance. Computed as the average over T perturbation steps of the indicator that the item remains in the top-K.
• NEG-P — range: [0, 1]
  • Analogous to POS-P but removes interactions in increasing order of importance. A higher score indicates that removing low-importance interactions has minimal effect, reflecting stronger explanation sufficiency.
• PN-S@K — range: [0, 1]
  • Probability of Necessity - Single. Measures the proportion of cases where the counterfactual explanation causes the target item to drop out of the top-K (item-level) or the ratio of items removed from the list (list-level).
• PN-R@K — range: [0, 1]
  • Position-aware extension of PN-S. Assigns higher weights to changes occurring at higher-ranked positions using a log-based denominator similar to NDCG, rewarding explanations that displace top-ranked items.
• Gini — range: [0, 1]
  • Measures explanation sparsity for implicit formats by quantifying the inequality of importance scores across interactions. Higher scores indicate explanations concentrate on fewer salient signals.
• #Perturb — range: other
  • Counts the exact number of user-item interactions changed by the counterfactual explanation in explicit formats. Lower values indicate higher sparsity.

Input / output format

Input: User interaction vector or user-item interaction graph, along with a target item or top-K recommendation list to be explained.

Output: Either importance scores for all interactions (implicit format) or a specific set of user-item interactions to perturb (explicit format).

Scoring recipe

def compute_pn_s(rank_original, rank_counterfactual, K):
    removed_count = 0
    for item in rank_original:
        if item not in rank_counterfactual:
            removed_count += 1
    return removed_count / K

def compute_pos_p(rank_original, rank_counterfactual_list, T, K):
    score = 0
    for t in range(T):
        cf_rank = rank_counterfactual_list[t]
        if rank_original[0] in cf_rank[:K]:
            score += 1
    return score / T

Common pitfalls

• Confusing item-level evaluation (generating K separate counterfactual explanations, one per target item) with list-level evaluation (generating a single unified explanation to clear the entire top-K list).
• Reversing the perturbation order for POS-P and NEG-P: POS-P must remove interactions in decreasing order of importance, while NEG-P uses increasing order.
• Applying the Gini Index without first normalizing importance scores via min-max scaling and sorting them in ascending order, which invalidates the sparsity calculation.

Evidence (verbatim from paper)

To quantify this property, we adopt two widely used perturbation-based metrics: Positive Perturbation (POS-P) and Negative Perturbation (NEG-P) (Barkan et al., 2024; Gurevitch et al., 2025; Mohammadi et al., 2025). Both metrics evaluate how quickly a target item drops out of the top-K list as input interactions are progressively masked. Specifically, POS-P removes interactions in decreasing order of importance, whereas NEG-P removes them in increasing order of importance.

Citation

@misc{nguyen2026from,
  title={From Top-1 to Top-K: A Reproducibility Study and Benchmarking of Counterfactual Explanations for Recommender Systems},
  author={Nguyen et al. (2026)},
  year={2026},
  note={arXiv:2604.19663}
}

• arXiv: 2604.19663