Helm Recommender Eval

helm-recommender-eval

HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems — Mehta (2026) (arXiv:2601.19197, 2026)

What this evaluates

Evaluates LLM-powered recommender systems across five human-centered dimensions: intent alignment, explanation quality, interaction naturalness, trust & transparency, and fairness & diversity. It combines expert human ratings with automated accuracy and fairness metrics across multiple domains and interaction scenarios.

Datasets

• MovieLens-1M — total 1000000; splits: test (-1)
• Amazon Books — total 287650; splits: test (-1)
• Yelp — total 612840; splits: test (-1)

Metrics

• Hit Rate@10 — range: [0, 1]
  • Fraction of ground-truth relevant items appearing in the top-10 recommendations.
• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality with logarithmic position discounting.
• Explanation Faithfulness — range: [0, 1]
  • Proportion of claimed item attributes in the LLM's explanation that can be verified against ground-truth metadata.
• Response Consistency — range: [0, 1]
  • Cosine similarity between model responses to paraphrased versions of the same query.
• Gini Coefficient — range: [0, 1]
  • Measures inequality in the frequency of recommended items across all scenarios; higher values indicate stronger popularity bias.
• Coverage@100 — range: [0, 1]
  • Proportion of unique catalog items that appear in the top-100 recommendations across all scenarios.
• Intra-List Diversity (ILD) — range: [0, 1]
  • Average pairwise dissimilarity between items within each recommendation list.

Input / output format

Input: Scripted user profile, interaction history, and scenario context (e.g., cold-start, preference refinement, contextual request) provided as prompts to the recommender system.

Output: Ranked list of recommended items (top-10 or top-100) accompanied by natural language explanations and multi-turn conversational responses.

Scoring recipe

def compute_metrics(predictions, gold, metadata, all_items):
    top10 = predictions[:10]
    hr = sum(1 for x in top10 if x in gold) / len(gold)
    dcg = sum(1/math.log2(i+2) for i, x in enumerate(top10) if x in gold)
    idcg = sum(1/math.log2(i+2) for i in range(len(gold)))
    ndcg = dcg/idcg if idcg > 0 else 0
    faithfulness = sum(1 for a in claimed_attrs if a in metadata) / len(claimed_attrs)
    consistency = cosine_sim(response, paraphrased_response)
    counts = Counter(predictions)
    gini = 1 - (2 * sum(counts.values() * (len(counts) - np.arange(len(counts)) - 1)) / (len(counts) * sum(counts.values())))
    coverage = len(set(predictions[:100])) / len(all_items)
    ild = np.mean([dissimilarity(i, j) for i, j in combinations(predictions[:100], 2)])
    return {'HR@10': hr, 'NDCG@10': ndcg, 'Faithfulness': faithfulness, 'Consistency': consistency, 'Gini': gini, 'Coverage@100': coverage, 'ILD': ild}

Common pitfalls

• Expert Likert scores (5-point scale) are aggregated per scenario but not combined into a single composite score, making cross-dimension comparison difficult.
• Explanation Faithfulness relies on external metadata availability; missing metadata can artificially lower faithfulness scores regardless of model capability.
• Gini Coefficient measures popularity bias across the entire evaluation set, not per-user fairness, which may misrepresent individual recommendation quality.

Evidence (verbatim from paper)

Complementing expert evaluation, we compute automated metrics: Hit Rate@10 and NDCG@10: Traditional accuracy metrics for baseline comparison. Explanation Faithfulness: Proportion of claimed item attributes verifiable against metadata. Response Consistency: Cosine similarity between responses to paraphrased queries. Gini Coefficient: Inequality in item recommendation frequency. Coverage@100: Proportion of catalog items appearing in the top-100 recommendations. Intra-List Diversity (ILD): Average pairwise dissimilarity within recommendation lists.

Citation

@misc{mehta2026helm,
  title={HELM: A Human-Centered Evaluation Framework for LLM-Powered Recommender Systems},
  author={Mehta (2026)},
  year={2026},
  note={arXiv:2601.19197}
}

• arXiv: 2601.19197