qhjqhj00 svld-points-ratio-eval
A Dataset and Benchmarks for Multimedia Social Analysis — Xue et al. (2020) (arXiv:2006.08335, 2020)
What this evaluates
Probes a model's ability to predict social engagement (upvote ratio) from multimodal inputs (images, videos, and text). It evaluates cross-modal fusion and regression capabilities on socially grounded, context-rich data.
Datasets
- SVLD — total 365336; splits: train (324888), val (40448), test (-1)
Metrics
Mean L1point ratio prediction error(primary) — range: [0, 1]- Mean Absolute Error (L1 loss) between the predicted points ratio and the ground-truth ratio (upvotes / total votes). Lower values indicate better regression performance.
Input / output format
Input: Per post: an image (resized to 224x224), a video (up to 64 frames sampled at ≤5 fps, padded with the last frame if needed), and three text fields (description, title, top comment) tokenized with BERT and padded/truncated to 102 tokens.
Output: A single scalar float representing the predicted points ratio (upvotes divided by total votes).
Scoring recipe
def compute_metric(predictions, gold):
errors = [abs(p - g) for p, g in zip(predictions, gold)]
return sum(errors) / len(errors)
Common pitfalls
- The test set is explicitly marked as 'unreleased', so external evaluation is not possible without author access.
- The metric is L1 error (lower is better), not accuracy or correlation; readers may mistakenly treat it as a classification score.
- Video preprocessing pads with the last frame when frames are insufficient, which can bias results if not replicated exactly.
Evidence (verbatim from paper)
Our goal in this baseline experiment is to model the points-ratio of a sample, the number of upvotes divided by the total number of votes. This is a number between zero and one, which roughly reflects the attitude of the community towards a particular post. Table 3 reports the Mean L1point ratio prediction error.
Citation
@misc{xue2020svld,
title={A Dataset and Benchmarks for Multimedia Social Analysis},
author={Xue et al. (2020)},
year={2020},
note={arXiv:2006.08335}
}
- arXiv: 2006.08335