Tvnf Negative Feedback Eval

tvnf-negative-feedback-eval

When Top-ranked Recommendations Fail: Modeling Multi-Granular Negative Feedback for Explainable and Robust Video Recommendation — Chen et al. (2025) (arXiv:2511.18700, 2025)

What this evaluates

Evaluates a model's ability to predict explicit and implicit user negative feedback for video recommendations, including binary judgment of controversial content and classification of dislike reasons. It also tests the model's capability to simulate user viewing behavior (e.g., fast-skip) based on historical interactions and profile data.

Datasets

• TVNF — total 2000; splits: train (-1), test (-1)
• MovieLens — total ?; splits: (unstated)
• Steam — total ?; splits: (unstated)

Metrics

• Recall (primary) — range: [0, 1]
  • Proportion of actual negative feedback instances correctly identified. Highlighted as the key metric for recognition rate of negative feedback videos.
• Binary Accuracy — range: [0, 1]
  • Proportion of correctly predicted binary judgments (negative vs. normal feedback).
• Precision — range: [0, 1]
  • Proportion of predicted negative feedback instances that are actually negative.
• F1_Score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall).
• Class_Acc — range: [0, 1]
  • Accuracy of feedback reason classification.
• Reasoning — range: [0, 1]
  • Relevance score between model-generated explanations and real user feedback reasons, assessed by GPT-4o on a 0-1 scale. Calculated only when the binary judgment is correct.
• Avg_Time — range: percent
  • Average watch time percentage in real-world deployment.
• Fast-skip Rate — range: percent
  • Percentage of recommended videos that users fast-skip.
• Dislike Rate — range: percent
  • Percentage of recommended videos that users dislike.

Input / output format

Input: For explicit feedback: video features (16 sampled images + title) and context. For implicit feedback: user profile, historical behavior sequences, and candidate video features.

Output: Binary judgment (negative/normal), predicted dislike reason category, and generated explanation/reasoning text.

Scoring recipe

def compute_metrics(pred_binary, gold_binary, pred_reason, gold_reason, model_explanation, gold_reason_text):
    binary_acc = sum(p == g for p, g in zip(pred_binary, gold_binary)) / len(gold_binary)
    tp = sum(1 for p, g in zip(pred_binary, gold_binary) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(pred_binary, gold_binary) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(pred_binary, gold_binary) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    class_acc = sum(p == g for p, g in zip(pred_reason, gold_reason)) / len(gold_reason)
    reasoning_scores = []
    for p_bin, g_bin, expl, gold_r in zip(pred_binary, gold_binary, model_explanation, gold_reason_text):
        if p_bin == 1 and g_bin == 1:
            reasoning_scores.append(gpt4o_evaluate_relevance(expl, gold_r))
    avg_reasoning = sum(reasoning_scores) / len(reasoning_scores) if reasoning_scores else 0
    return binary_acc, precision, recall, f1, class_acc, avg_reasoning

Common pitfalls

• Recall is emphasized as the key metric for explicit feedback recognition, but high recall may come at the cost of lower precision.
• The reasoning score is only computed when the initial binary judgment is correct, which can mask poor explanation quality for misclassified cases.
• Implicit feedback prediction suffers from high noise and randomness, making direct comparison with explicit feedback metrics misleading without context.

Evidence (verbatim from paper)

Evaluation metrics include binary accuracy, precision, recall, and F1-score for negative feedback prediction, as well as the accuracy of feedback reason classification. For explicit negative feedback, we employ GPT-4o to assess the relevance between the model’s explanations and users’ real feedback reasons, with relevance scores ranging from 0 to 1.

Citation

@misc{chen2025when,
  title={When Top-ranked Recommendations Fail: Modeling Multi-Granular Negative Feedback for Explainable and Robust Video Recommendation},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.18700}
}

• arXiv: 2511.18700