hit-ratio-eval
Not Just What, But When: Integrating Irregular Intervals to LLM for Sequential Recommendation — Du et al. (2025) (arXiv:2507.23209, 2025)
What this evaluates
Evaluates the ability of LLM-based sequential recommendation models to predict the next item in a user's interaction history. It specifically probes how well models capture temporal dynamics by incorporating irregular time intervals between interactions, and assesses performance under warm and cold-start conditions.
Datasets
- Amazon Reviews (Video Games, CDs and Vinyl, Books) — total ?; splits: train (-1), val (-1), test (-1)
Metrics
Hit Ratio@1 (primary) — range: percent
- Hit Ratio@1 measures the proportion of users for whom the ground-truth next item appears in the model's top-1 prediction. It is calculated as the number of correct top-1 predictions divided by the total number of test instances.
Input / output format
Input: User interaction history sequences containing item identifiers and irregular time intervals, formatted into an optionalized text prompt for an LLM (LLaMA-2 7B).
Output: A single predicted item identifier/name from the candidate set.
Scoring recipe
def hit_ratio_at_1(predictions, golds):
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
return correct / len(golds)
Common pitfalls
- Using raw timestamps as text prompts instead of computed intervals, which the ablation study shows degrades performance.
- Ignoring the validation ratio metric, which tracks invalid LLM outputs (e.g., word mismatches with the candidate set), though it was 100% here due to prompt design.
- Misdefining warm/cold scenarios; they are based on the top/bottom 35% of interaction frequency or average interval length, not standard train/test splits.
Evidence (verbatim from paper)
Evaluation Metric. We follow previous work (Liao et al., [2024]) to use Hit Ratio@1. Notably, (Liao et al., [2024]) further adds the validation ratio as an additional metric, as certain models may generate invalid responses, such as word mismatches with the candidate set; however, in our method, the validation ratio is consistently 100% attributed to our optionalized prompt design. We adopt a leave-one-out evaluation strategy following previous work (Kang and McAuley, [2018]), where the last item in each user’s sequence is reserved for testing, the second-to-last item for validation, and the remaining items are used for training.
Citation
@misc{du2025interval,
title={Not Just What, But When: Integrating Irregular Intervals to LLM for Sequential Recommendation},
author={Du et al. (2025)},
year={2025},
note={arXiv:2507.23209}
}
1---2name: hit-ratio-eval3description: Evaluates the ability of LLM-based sequential recommendation models to predict the next item in a user's interaction history. It specifically probes how well models capture temporal dynamics by incorporating irregular time intervals between interactions, and assesses performance under warm and cold-start conditions. Use when the user wants to benchmark on Amazon Reviews (Video Games, CDs and Vinyl, Books), or asks about evaluating this task. Reports Hit Ratio@1.4---56# hit-ratio-eval78> Not Just What, But When: Integrating Irregular Intervals to LLM for Sequential Recommendation — Du et al. (2025) (arXiv:2507.23209, 2025)910## What this evaluates1112Evaluates the ability of LLM-based sequential recommendation models to predict the next item in a user's interaction history. It specifically probes how well models capture temporal dynamics by incorporating irregular time intervals between interactions, and assesses performance under warm and cold-start conditions.1314## Datasets1516- **Amazon Reviews (Video Games, CDs and Vinyl, Books)** — total ?; splits: train (-1), val (-1), test (-1)1718## Metrics1920- `Hit Ratio@1` **(primary)** — range: percent21 - Hit Ratio@1 measures the proportion of users for whom the ground-truth next item appears in the model's top-1 prediction. It is calculated as the number of correct top-1 predictions divided by the total number of test instances.2223## Input / output format2425**Input**: User interaction history sequences containing item identifiers and irregular time intervals, formatted into an optionalized text prompt for an LLM (LLaMA-2 7B).2627**Output**: A single predicted item identifier/name from the candidate set.2829## Scoring recipe3031```python32def hit_ratio_at_1(predictions, golds):33 correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)34 return correct / len(golds)35```3637## Common pitfalls3839- Using raw timestamps as text prompts instead of computed intervals, which the ablation study shows degrades performance.40- Ignoring the validation ratio metric, which tracks invalid LLM outputs (e.g., word mismatches with the candidate set), though it was 100% here due to prompt design.41- Misdefining warm/cold scenarios; they are based on the top/bottom 35% of interaction frequency or average interval length, not standard train/test splits.4243## Evidence (verbatim from paper)4445> Evaluation Metric. We follow previous work (Liao et al., [2024]) to use Hit Ratio@1. Notably, (Liao et al., [2024]) further adds the validation ratio as an additional metric, as certain models may generate invalid responses, such as word mismatches with the candidate set; however, in our method, the validation ratio is consistently 100% attributed to our optionalized prompt design. We adopt a leave-one-out evaluation strategy following previous work (Kang and McAuley, [2018]), where the last item in each user’s sequence is reserved for testing, the second-to-last item for validation, and the remaining items are used for training.4647## Citation4849```bibtex50@misc{du2025interval,51 title={Not Just What, But When: Integrating Irregular Intervals to LLM for Sequential Recommendation},52 author={Du et al. (2025)},53 year={2025},54 note={arXiv:2507.23209}55}56```5758- arXiv: 2507.23209
qhjqhj00