# Samrec Eval

> Evaluates sequential recommendation models on their ability to predict the next item in a user's chronological interaction history. It specifically probes whether sharpness-aware minimization improves generalization and data efficiency compared to standard Transformers and self-supervised baselines. Use when the user wants to benchmark on Amazon-Beauty, Amazon-Sports, Amazon-Toys, Yelp, or asks about evaluating this task. Reports HR@10.

- Skill: `qhjqhj00/samrec-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/samrec-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/samrec-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/samrec-eval

---


# samrec-eval

> Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation — Lai et al. (2023) (arXiv:2308.10347, 2023)

## What this evaluates

Evaluates sequential recommendation models on their ability to predict the next item in a user's chronological interaction history. It specifically probes whether sharpness-aware minimization improves generalization and data efficiency compared to standard Transformers and self-supervised baselines.

## Datasets

- **Amazon-Beauty** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Sports** — total ?; splits: train (-1), val (-1), test (-1)
- **Amazon-Toys** — total ?; splits: train (-1), val (-1), test (-1)
- **Yelp** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `HR@10` **(primary)** — range: [0, 1]
  - 1 if the ground-truth item appears in the top-10 predicted items, else 0.
- `NDCG@10` — range: [0, 1]
  - Sum of 1/log2(rank+1) for each hit in the top-10 list, normalized by the ideal DCG@10 (which equals 1 for a single positive item).

## Input / output format

**Input**: A chronological sequence of user-item interaction IDs.

**Output**: A ranked list of top-10 candidate item IDs.

## Scoring recipe

```python
def evaluate(predictions, ground_truth):
    hr = 1.0 if ground_truth in predictions else 0.0
    if ground_truth in predictions:
        rank = predictions.index(ground_truth) + 1
        ndcg = 1.0 / math.log2(rank + 1)
    else:
        ndcg = 0.0
    return hr, ndcg
```

## Common pitfalls

- Failing to apply the 5-core filtering before splitting the data.
- Not sorting user interactions by timestamp before creating train/val/test splits.
- Reporting results from a single run instead of averaging over 10 independent runs.

## Evidence (verbatim from paper)

> We use two widely used metrics, Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG), to evaluate performance. We repeat the experiments 10 times independently and report the average results for HR@10 and NDCG@10.

## Citation

```bibtex
@misc{lai2023enhancing,
  title={Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation},
  author={Lai et al. (2023)},
  year={2023},
  note={arXiv:2308.10347}
}
```

- arXiv: 2308.10347