Vidas Eval

vidas-eval

ViDAS: Vision-based Danger Assessment and Scoring — Gupta et al. (2024) (arXiv:2410.00477, 2024)

What this evaluates

Evaluates a model's ability to assess danger levels in videos by identifying risk elements, understanding context, and assigning severity scores. It probes multimodal perception and risk reasoning capabilities.

Datasets

• ViDAS — total 100; splits: test (100)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between predicted danger ratings and average human ratings. Formula: MSE = (1/n) * Σ(L_i - E_i^(avg))^2, where L_i is the LLM's predicted rating, E_i^(avg) is the average human rating, and n is the number of videos.

Input / output format

Input: Video(s) accompanied by a prompt template (Zero-Shot, Fixed Few-Shot, or N-shot learning) requesting a danger level assessment.

Output: A single numerical danger rating score on a 0–10 scale.

Scoring recipe

def compute_mse(predictions, gold_ratings):
    n = len(predictions)
    mse = sum((p - g) ** 2 for p, g in zip(predictions, gold_ratings)) / n
    return mse

Common pitfalls

• The metric uses average human ratings per video, not individual annotator scores.
• Performance depends heavily on prompt type (ZS vs FFS vs N-shot) and example selection, not just N value.
• LLMs may fail to detect hidden danger elements unless explicitly pointed out in the prompt or video context.

Evidence (verbatim from paper)

To compare the danger ratings predicted by a Language Model (LLM) to the average danger ratings given by humans, we use the Mean Squared Error (MSE) as our metric. The MSE provides a measure of the average squared difference between the predicted ratings and the actual average ratings. Let $L_{i}$ represent the danger rating predicted by the LLM for the $i$-th video, and let $E_{i}^{( ext{avg})}$ denote the average danger rating given by human evaluators for the $i$-th video. The MSE can be formulated as follows: $\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(L_{i}-E_{i}^{(\text{avg})})^{2}$ where $n$ is the total number of videos. The MSE thus quantifies the prediction accuracy by averaging the squared differences between the predicted and actual ratings overall videos.

Citation

@misc{gupta2024vidas,
  title={ViDAS: Vision-based Danger Assessment and Scoring},
  author={Gupta et al. (2024)},
  year={2024},
  note={arXiv:2410.00477}
}

• arXiv: 2410.00477