Vimrhp Eval

vimrhp-eval

ViMRHP: A Vietnamese Benchmark Dataset for Multimodal Review Helpfulness Prediction via Human-AI Collaborative Annotation — Nguyen et al. (2025) (arXiv:2505.07416, 2025)

What this evaluates

Evaluates a model's ability to predict the helpfulness of product reviews by jointly processing textual descriptions and visual content. It probes multimodal alignment and ranking capabilities in low-resource language settings, specifically Vietnamese.

Datasets

• ViMRHP — total 46000; splits: unspecified (-1); repo https://github.com/trng28/ViMRHP

Metrics

• MAP — range: [0, 1]
  • Mean Average Precision: computes the average precision across all queries, averaging precision values at ranks where relevant items occur.
• NDCG@K (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff K: measures ranking quality by discounting gains logarithmically with rank position and normalizing by the ideal DCG. Evaluated at K ∈ {1, 3, 5}.

Input / output format

Input: Product description, review text, and associated product/review images.

Output: A predicted helpfulness score or ranking score per review.

Scoring recipe

def compute_ndcg_at_k(pred_scores, true_labels, k):
    ranked_idx = np.argsort(-pred_scores)
    true_ranked = [true_labels[i] for i in ranked_idx[:k]]
    dcg = sum(l / np.log2(i + 2) for i, l in enumerate(true_ranked))
    ideal = sorted(true_labels, reverse=True)[:k]
    idcg = sum(l / np.log2(i + 2) for i, l in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

def compute_map(pred_scores, true_labels):
    precisions = []
    rel_count = 0
    for i, score in enumerate(np.argsort(-pred_scores)):
        if true_labels[score] > 0:
            rel_count += 1
            precisions.append(rel_count / (i + 1))
    return np.mean(precisions) if precisions else 0.0

Common pitfalls

• Evaluating at K values other than 1, 3, or 5, which the authors explicitly tie to user reading behavior.
• Ignoring the multimodal requirement by using text-only baselines or dropping image features, despite the dataset's core contribution.
• Failing to apply the MatchZoo threshold of 3.0 when computing MAP/NDCG, which affects how relevance is binarized or graded.

Evidence (verbatim from paper)

Following the study by Liu et al. [[4]] on the Amazon-MRHP and Lazada-MRHP datasets, we adopt two evaluation metrics frequently used in recommendation: MAP (Mean Average Precision) and NDCG@K (Normalized Discounted Cumulative Gain), where $K\in{1,3,5}$, to assess the ViMRHP dataset.

Citation

@misc{nguyen2025vimrhp,
  title={ViMRHP: A Vietnamese Benchmark Dataset for Multimodal Review Helpfulness Prediction via Human-AI Collaborative Annotation},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2505.07416}
}

• arXiv: 2505.07416