Llara Eval

llara-eval

LLaRA: Large Language-Recommendation Assistant — Jiayi Liao et al. (2023) (arXiv:2312.02445, 2023)

What this evaluates

Evaluates a model's ability to predict the next item in a user's sequential interaction history by leveraging both semantic item metadata and behavioral embeddings. It also measures the model's instruction-following capability in generating valid recommendations from a candidate set.

Datasets

• MovieLens100K — total 943; splits: train (-1), val (-1), test (-1); repo https://github.com/ljy0ustc/LLaRA
• Steam — total 11938; splits: train (-1), val (-1), test (-1); repo https://github.com/ljy0ustc/LLaRA

Metrics

• HitRatio@1 (primary) — range: [0, 1]
  • Proportion of test sequences where the model's top-1 predicted item matches the ground truth next item, evaluated over a candidate set of 20 items (1 positive + 19 negatives).
• valid ratio — range: [0, 1]
  • Proportion of generated predictions that are valid items belonging to the candidate set, measuring the model's instruction-following and output validity.

Input / output format

Input: Textual prompt containing user interaction history (last 10 items, padded if necessary), a candidate set of 20 items (1 ground truth + 19 negatives), and a placeholder for the next item. Items are represented using a hybrid of behavioral tokens and textual titles.

Output: A single item name or identifier predicted by the model as the next interaction.

Scoring recipe

def score(predictions, golds, candidates):
    hit_ratio = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    valid_ratio = sum(1 for p in predictions if p in candidates) / len(predictions)
    return {'HitRatio@1': hit_ratio, 'valid ratio': valid_ratio}

Common pitfalls

• LLMs may generate out-of-vocabulary or nonsensical tokens, resulting in invalid predictions that must be filtered or penalized via the valid ratio metric.
• Traditional sequential recommenders output probability distributions over all items; they must be explicitly adapted to select only from the provided candidate set for fair comparison.
• Chronological splitting is critical; failing to exclude future interactions from training data causes severe information leakage and inflated metrics.

Evidence (verbatim from paper)

For each sequence, we randomly select 20 non-interacted items to construct the candidate set, ensuring the inclusion of the correct subsequent item. LLaRA and other baseline models aim to identify the correct item from this candidate set, and their performance is evaluated using the HitRatio@1 metric. With appropriate prompting, LLM-based recommenders can generate a single candidate item as required. To adapt traditional models to this setting, we select the item with the highest probability in the candidate set as the prediction. Meanwhile, since LLaRA employs a generative paradigm for prediction, which may yield invalid responses such as nonsensical words or items outside the candidate sets, we introduce an additional metric — valid ratio. It quantifies the proportion of valid responses (i.e., items in the candidate set) across all sequences.

Citation

@misc{liao2023llara,
  title={LLaRA: Large Language-Recommendation Assistant},
  author={Jiayi Liao et al. (2023)},
  year={2023},
  note={arXiv:2312.02445}
}

• arXiv: 2312.02445