Mem Rec Ctr Eval

mem-rec-ctr-eval

Mem-Rec: Memory Efficient Recommendation System using Alternative Representation — Gopi Krishna Jha et al. (2023) (arXiv:2305.07205, 2023)

What this evaluates

Evaluates the click-through rate prediction capability of deep learning recommendation models under varying memory budgets and compression techniques. It probes how well alternative representation schemes (like Bloom filter encoding) maintain recommendation accuracy while drastically reducing embedding table sizes.

Datasets

• Avazu — total 40000000; splits: train (-1), val (-1), test (-1)
• Criteo-Kaggle — total 46000000; splits: train (-1), val (-1), test (-1)
• Criteo-Terabyte — total 430000000; splits: train (-1), val (-1), test (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Receiver Operating Characteristics-Area Under The Curve. Measures the probability that a randomly chosen positive instance (click) is ranked higher than a randomly chosen negative instance (no click). Computed over the test set predictions and ground truth labels.

Input / output format

Input: A record containing numeric and categorical features describing an online advertisement served to a user.

Output: A predicted click probability (scalar between 0 and 1).

Scoring recipe

def compute_auc(predictions, labels):
    # predictions: array of predicted click probabilities
    # labels: array of ground truth binary click labels (0 or 1)
    # Compute ROC-AUC over the test set
    auc = 0.0
    for i in range(len(labels)):
        for j in range(len(labels)):
            if labels[i] > labels[j]:
                auc += 1.0
            elif labels[i] == labels[j]:
                auc += 0.5
    auc /= (sum(labels) * sum(1 - l for l in labels))
    return auc

Common pitfalls

• AUC changes as small as 0.001 are considered significant in commercial settings, so reporting only 2 decimal places hides meaningful differences.
• Datasets are split chronologically by day (first n-1 days for training, nth day for validation/test), not randomly, which affects generalization evaluation and requires strict temporal ordering during data loading.

Evidence (verbatim from paper)

Metrics: Consistent with prior work, we use ROC-AUC (Receiver Operating Characteristics-Area Under The Curve) to determine the quality of recommendation generated by our model (Desai et al., [2022]). One important point to note here is that in the case of commercial-scale recommender systems, even a 0.001 change in AUC has a significant business impact (Desai et al., [2021]).

Citation

@misc{jha2023memrec,
  title={Mem-Rec: Memory Efficient Recommendation System using Alternative Representation},
  author={Gopi Krishna Jha et al. (2023)},
  year={2023},
  note={arXiv:2305.07205}
}

• arXiv: 2305.07205