Videoconviction Eval

videoconviction-eval

VideoConviction: A Multimodal Benchmark for Human Conviction and Stock Market Recommendations — Galarnyk et al. (2025) (arXiv:2507.08104, 2025)

What this evaluates

Evaluates whether LLMs and MLLMs can accurately extract stock tickers, identify explicit investment actions, and quantify human conviction levels from financial influencer videos and transcripts. It probes multimodal reasoning, financial domain understanding, and the ability to filter out noisy or promotional content.

Datasets

• VideoConviction — total 6063; splits: test (-1); repo https://github.com/gtfintechlab/VideoConviction

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard F1 score computed as the harmonic mean of precision and recall for each task (T, TA, TAC).

Input / output format

Input: Full-length video, segmented video, full-length transcript, or segmented transcript.

Output: Predicted ticker name, investment action, and conviction score extracted simultaneously from a single prompt.

Scoring recipe

# For each task (T, TA, TAC):
# predictions = parse_model_output(raw_response)
# gold = human_annotation
# precision = len(set(predictions) & set(gold)) / len(set(predictions))
# recall = len(set(predictions) & set(gold)) / len(set(gold))
# f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
# return f1

Common pitfalls

• Models frequently misclassify general market commentary as explicit buy/sell recommendations.
• Full-length video inputs introduce significant noise (sponsorships, unrelated stock discussions) that degrades performance compared to segmented inputs.
• Open-source MLLMs are constrained by short context windows (≤4 minutes), limiting their ability to process full videos.

Evidence (verbatim from paper)

Performance across all three tasks is evaluated using the F1 score metric.

Citation

@misc{galarnyk2025videoconviction,
  title={VideoConviction: A Multimodal Benchmark for Human Conviction and Stock Market Recommendations},
  author={Galarnyk et al. (2025)},
  year={2025},
  note={arXiv:2507.08104}
}

• arXiv: 2507.08104