Star Agqa QA Eval

star-agqa-qa-eval

DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes — Linok et al. (2025) (arXiv:2505.03581, 2025)

What this evaluates

Evaluates a model's ability to perform spatio-temporal reasoning and answer questions about dynamic scenes using compressed textual scene graph sequences. It probes the model's capacity to track object interactions, understand event ordering, and generalize to unseen temporal compositions without relying on raw visual inputs.

Datasets

• STAR — total 60000; splits: validation (-1)
• AGQA — total 2270000; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Prediction is considered correct if the dataset ground truth answer contains the response generated by the model. Reported as the fraction of correct predictions over the total.
• BLEU — range: [0, 1]
  • N-gram overlap precision between generated answer and ground truth, typically with geometric smoothing.
• METEOR — range: [0, 1]
  • Weighted n-gram match score considering synonyms, stemming, and word order alignment.
• BERTScore — range: [0, 1]
  • Cosine similarity between contextual embeddings of generated and reference answers, aggregated via F1.

Input / output format

Input: A prompt containing a question (Q) and a compressed latent representation of a sequence of textual scene graphs, formatted as: 'Based on scene graph, h_llm, Q'.

Output: A natural language answer string corresponding to the question.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # Handle multi-answer ground truth if present
        if isinstance(gold, list):
            gold = gold[0]
        # Substring match as specified in protocol
        if pred.lower() in gold.lower():
            correct += 1
    return correct / len(predictions)

Common pitfalls

• The benchmark explicitly requires processing textual scene graph sequences, not raw video frames; feeding visual inputs violates the protocol.
• STAR results are reported on the validation split, while AGQA results are reported on the test split; mixing these splits will cause score mismatches.
• Ground truth answers in AGQA are multi-label/multi-phrase; the evaluation checks if the model's single generated answer is contained within any of the valid answers.

Evidence (verbatim from paper)

To evaluate answer quality, we use Accuracy as a primary metric to be in alignment with previous research, borrowing metrics from the corresponding publications. Under this metric, our prediction is considered correct if the dataset ground true answer contains the response generated by the model.

Citation

@misc{linok2025dygenc,
  title={DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenes},
  author={Linok et al. (2025)},
  year={2025},
  note={arXiv:2505.03581}
}

• arXiv: 2505.03581