Sign Recommender Eval

sign-recommender-eval

Detecting Beneficial Feature Interactions for Recommender Systems — Su et al. (2020) (arXiv:2008.00404, 2020)

What this evaluates

This protocol evaluates a model's ability to detect beneficial feature interactions for recommendation and graph classification tasks. It measures prediction accuracy and ranking performance while assessing how well the model filters out irrelevant feature pairs to improve generalization.

Datasets

• Frappe — total 288609; splits: train (-1), val (-1), test (-1)
• MovieLens-tag — total 2006859; splits: train (-1), val (-1), test (-1)
• Twitter — total 144033; splits: train (-1), val (-1), test (-1)
• DBLP — total 19456; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (ACC) (primary) — range: [0, 1]
  • Correct predictions divided by total predictions. A threshold of 0.5 is applied to continuous scores to derive binary labels for classification.
• area under a curve with Riemann sums (AUC) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, explicitly computed using Riemann sums rather than the standard trapezoidal rule.

Input / output format

Input: Graph-structured data where nodes represent entities (e.g., user, item, context/tag) and edges represent feature interactions. Node features are embedded vectors.

Output: Continuous interaction score (for recommender datasets) or discrete class label (for graph classification datasets).

Scoring recipe

def compute_metrics(pred_scores, gold_labels):
    acc = sum(1 for p, g in zip(pred_scores, gold_labels) if (p > 0.5) == g) / len(gold_labels)
    # AUC computed via Riemann sums as specified in the paper
    auc = compute_riemann_auc(gold_labels, pred_scores)
    return {'acc': acc, 'auc': auc}

Common pitfalls

• The paper uses a non-standard 70/15/15 train/val/test split instead of the more common 80/10/10 or 60/20/20 splits.
• AUC is explicitly calculated using Riemann sums rather than the standard trapezoidal rule or default library implementations.
• The same metric names (ACC and AUC) are applied to both recommendation (edge prediction) and graph classification tasks, which may cause confusion when comparing results across different dataset types.

Evidence (verbatim from paper)

Each dataset is randomly split into training, validation, and test datasets with a proportion of 70%, 15%, and 15%. We choose the model parameters that produce the best results in validation set when the number of predicted edges being steady. We use accuracy (ACC) and the area under a curve with Riemann sums (AUC) as evaluation metrics.

Citation

@misc{su2020detecting,
  title={Detecting Beneficial Feature Interactions for Recommender Systems},
  author={Su et al. (2020)},
  year={2020},
  note={arXiv:2008.00404}
}

• arXiv: 2008.00404