Tsg Bench Eval

tsg-bench-eval

LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study — Dongil Yang et al. (2025) (arXiv:2505.19510, 2025)

What this evaluates

Evaluates large language models' ability to understand and generate structured scene graphs from textual narratives. It probes spatial reasoning, action decomposition, and the capacity to map dynamic descriptions to discrete visual or structural elements.

Datasets

• TSG Bench — total ?; splits: test (-1); repo https://github.com/docworlds/tsg-bench

Metrics

• Exact Match (EM) / Accuracy (primary) — range: [0, 1]
  • Accuracy is calculated as the proportion of instances where the model's predicted single-letter candidate exactly matches the reference answer. EM requires exact string matching without tolerance for formatting or case differences.
• Precision, Recall, Macro F1 (primary) — range: [0, 1]
  • Precision = TP / (TP + FP), Recall = TP / (TP + FN), Macro F1 = average of per-class F1 scores. For MA-SGG, each generated graph is evaluated separately against its reference before aggregation.

Input / output format

Input: Textual narrative descriptions. For SGQA, inputs include the narrative, a question, and multiple-choice candidates. For SGG (SA-SGG and MA-SGG), inputs consist solely of the narrative text.

Output: For SGQA: a single uppercase letter corresponding to the chosen candidate. For SGG: a structured scene graph representation (nodes and edges) describing the scene(s) extracted from the narrative.

Scoring recipe

def score_sgqa(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().upper() == g.strip().upper())
    return correct / len(golds)

def score_sgg(predictions, golds):
    tp, fp, fn = 0, 0, 0
    for preds, gold in zip(predictions, golds):
        graphs = preds if isinstance(preds, list) else [preds]
        for pred_graph in graphs:
            pred_edges = set(pred_graph.edges)
            gold_edges = set(gold.edges)
            tp += len(pred_edges & gold_edges)
            fp += len(pred_edges - gold_edges)
            fn += len(gold_edges - pred_edges)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return prec, rec, f1

Common pitfalls

• In MA-SGG, failing to evaluate each generated graph separately instead of aggregating them first, which artificially inflates precision and recall.
• Assuming high precision implies complete scene coverage; the paper notes models often show higher precision than recall due to incomplete decomposition of sub-scenes.
• Treating zero-shot prompting as sufficient for complex temporal decomposition; performance drops significantly without chain-of-thought or error-guided refinement.

Evidence (verbatim from paper)

We assess SGQA using Exact Match (EM), which requires the model’s generation to match the reference element exactly. For SGQA, we instruct LLMs to generate a single letter representing the predicted candidate and evaluate it using accuracy. Scene graph generation tasks are assessed with precision, recall, and macro F1 score. For MA-SGG, where one description yields multiple scene graphs, evaluation is conducted separately for each generated graph.

Citation

@misc{yang2025tsgbench,
  title={LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study},
  author={Dongil Yang et al. (2025)},
  year={2025},
  note={arXiv:2505.19510}
}

• arXiv: 2505.19510