Malloc Eval

malloc-eval

MALLOC: Benchmarking the Memory-aware Long Sequence Compression for Large Sequential Recommendation — Yu et al. (2026) (arXiv:2601.20234, 2026)

What this evaluates

Evaluates memory-aware long sequence compression techniques in large-scale sequential recommendation. It probes how well methods balance memory overhead, computational cost, and ranking accuracy when processing long user interaction histories to predict future clicks.

Datasets

• Amazon-Electronic — total 3333478; splits: train (-1), val (-1), test (-1)
• MicroVideo1.7M — total 13135234; splits: train (-1), val (-1), test (-1)
• KuaiVideo — total 10869055; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Probability that a positive instance is ranked higher than a negative one: AUC = (1/|P|) * sum_{(u,v+,v-) in P} I(y_hat_u,v+ > y_hat_u,v-).
• GAUC — range: [0, 1]
  • Grouped AUC: weighted average of per-user AUC scores, where weights are the number of impressions per user: GAUC = sum_{u in U} (N_u * AUC_u) / sum_{u in U} N_u.
• Logloss — range: [0, ∞)
  • Binary cross-entropy over the test set: -1/|D| * sum_{(u,v) in D} (y_u,v * log(y_hat_u,v) + (1-y_u,v) * log(1-y_hat_u,v)).
• MACs — range: other
  • Count of multiply-accumulate operations performed by the model during a single inference step.
• Memory Occupation — range: other
  • Total amount of memory (e.g., in GB) required to store hidden states and KV cache during inference.

Input / output format

Input: User interaction sequence S_u (ordered list of item IDs) and a candidate item v.

Output: Predicted interaction probability/score y_hat_u,v.

Scoring recipe

def compute_auc(gold_pos, gold_neg):
    return sum(1 for p in gold_pos for n in gold_neg if p > n) / (len(gold_pos) * len(gold_neg))

def compute_gauc(user_preds, user_gold, user_impressions):
    user_aucs = [compute_auc([p for p in user_preds[u] if p in user_gold[u]],
                              [p for p in user_preds[u] if p not in user_gold[u]]) for u in user_preds]
    return sum(user_impressions[u] * a for u, a in enumerate(user_aucs)) / sum(user_impressions)

def compute_logloss(y_true, y_pred):
    return -mean(y_true * log(y_pred) + (1 - y_true) * log(1 - y_pred))
# MACs and Memory Occupation are measured via hardware profiler during inference

Common pitfalls

• Confusing MACs (compute operations) with Memory Occupation (storage overhead), as both are resource metrics but measure fundamentally different bottlenecks.
• Using standard AUC instead of GAUC can mask performance degradation on low-activity users, since GAUC weights by per-user impression counts.
• Assuming compression methods preserve the original sequence length; many prune or merge tokens, altering the effective input length for the backbone.

Evidence (verbatim from paper)

Specifically, MALLOC introduces three metrics, AUC, GAUC, and Logloss for recommendation evaluation, and another two metrics, MACs, and memory occupation for resource calculation. AUC (Area Under the ROC Curve) measures the probability that a positive instance is ranked higher than a negative one. GAUC (Grouped AUC) extends AUC by computing it separately for each user and then calculating the weighted average across all users. MACs (Multiply–Accumulate Operations) represent the number of elementary multiply-and-add operations performed by a model during inference. Memory Occupation measures the amount of memory for storing the hidden states.

Citation

@misc{yu2026malloc,
  title={MALLOC: Benchmarking the Memory-aware Long Sequence Compression for Large Sequential Recommendation},
  author={Yu et al. (2026)},
  year={2026},
  note={arXiv:2601.20234}
}

• arXiv: 2601.20234