Od LLM Eval

od-llm-eval

On-Device Large Language Models for Sequential Recommendation — Xin Xia et al. (2026) (arXiv:2601.09306, 2026)

What this evaluates

Evaluates the ability of compressed large language models to perform sequential recommendation on resource-constrained devices. It probes how well a model preserves ranking quality and recommendation accuracy after aggressive parameter compression (SVD + normalization) while maintaining low latency.

Datasets

• Amazon Instruments, Games, Arts — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• HR@5 — range: [0, 1]
  • Hit Ratio at cutoff K: fraction of test users for whom the true next item appears in the top-K predicted list.
• HR@10 — range: [0, 1]
  • Hit Ratio at cutoff K: fraction of test users for whom the true next item appears in the top-K predicted list.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff K: weighted sum of relevance scores (1 for hit) discounted by rank position, normalized by the ideal DCG.
• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff K: weighted sum of relevance scores (1 for hit) discounted by rank position, normalized by the ideal DCG.

Input / output format

Input: User interaction history sequences represented as item titles and descriptions, formatted as prompts for the LLM.

Output: A ranked list of top-K candidate items (K=5 or 10) generated by the model.

Scoring recipe

def evaluate(gold_item, pred_list, k):
    top_k = pred_list[:k]
    hr = 1.0 if gold_item in top_k else 0.0
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(top_k) if item == gold_item)
    idcg = 1.0 / math.log2(2)
    ndcg = dcg / idcg
    return hr, ndcg

Common pitfalls

• The validation and test sets are constructed deterministically using the last two interactions per user sequence, not via random or temporal splits.
• Main results assume a fixed 50% compression ratio; varying this ratio significantly impacts HR/NDCG scores.
• Calibration set size (256 samples) is fixed for the primary evaluation; changing it alters the one-shot SVD decomposition quality.

Evidence (verbatim from paper)

For any sequence $S_{u}={v_{1}^{u},v_{2}^{u},\ldots,v_{\ell}^{u}}$, the last two interacting items, $v_{\ell}^{u},v_{\ell-1}^{u}$, are used as the validation and test set respectively. Since our method is designed for sequential recommendation tasks, we use Hit Ratio and NDCG (Normalized Discounted Cumulative Gain) to evaluate the recommendation results of the top-5 and top-10 results.

Citation

@misc{xia2026ondevice,
  title={On-Device Large Language Models for Sequential Recommendation},
  author={Xin Xia et al. (2026)},
  year={2026},
  note={arXiv:2601.09306}
}

• arXiv: 2601.09306