Factual Scene Graph Parsing Eval

factual-scene-graph-parsing-eval

FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing — Li et al. (2023) (arXiv:2305.17497, 2023)

What this evaluates

This benchmark evaluates a model's ability to parse natural language captions into structured scene graphs that faithfully represent described visual elements. It probes compositional generalization and output consistency by testing parsers on both standard and length-constrained splits, measuring how well generated graph structures align with human-annotated ground truth.

Datasets

• FACTUAL — total 40369; splits: train (37861), val (1000), test (1508); repo https://github.com/zhuang-li/FACTUAL

Metrics

• SPICE (primary) — range: [0, 1]
  • Computes the F-score based on the overlap of sub-components (objects, {object, attribute} tuples, and {object, predicate, object} triples) between the candidate and ground truth scene graphs.
• Exact Set Match — range: [0, 1]
  • Calculates the percentage of instances where the set of parsed fact strings exactly matches the ground truth fact strings, ignoring the order of facts.
• SoftSPICE — range: [0, 1]
  • Embedding-based graph similarity that encodes graph sub-components using Sentence-BERT and computes the average cosine similarity between candidate and reference component embeddings.

Input / output format

Input: Natural language captions describing image regions.

Output: Structured scene graphs represented as sets of triples (object, predicate, object) and tuples (object, attribute), formatted according to the target annotation scheme (e.g., FACTUAL-SG, CDP-SG, or VG-SG).

Scoring recipe

def exact_set_match(pred_graph, gold_graph):
    pred_facts = set(str(f) for f in pred_graph)
    gold_facts = set(str(f) for f in gold_graph)
    if not gold_facts: return 0.0
    return len(pred_facts & gold_facts) / len(gold_facts)

def spice_score(pred_graph, gold_graph):
    pred_comps = extract_components(pred_graph)
    gold_comps = extract_components(gold_graph)
    matches = len(pred_comps & gold_comps)
    p = matches / max(len(pred_comps), 1)
    r = matches / max(len(gold_comps), 1)
    return 2 * p * r / (p + r) if (p + r) > 0 else 0.0

Common pitfalls

• SPICE-Parser uses a dependency parser trained on general domain text rather than the specific dataset's annotation scheme, causing systematic mismatches with ground truth graphs.
• Exact Set Match ignores fact ordering, potentially masking structural parsing errors where correct facts are arranged in an invalid graph topology.
• Human evaluation of faithfulness and consistency relies on a small panel of three students, introducing subjectivity and limited statistical power.

Evidence (verbatim from paper)

Following Schuster et al. (2015); Wang et al. (2018); Choi et al. (2022), we evaluate scene graph parsers utilizing the SPICE metric (Anderson et al. (2016)). The SPICE F-score measures the similarity between the candidate and ground truth graph representations extracted from captions by the parsers. In addition, we also employ the Exact Set Match metric (Yu et al. (2019)), which assesses the accuracy of the parsers by determining whether the strings of the parsed facts match the ground truth facts while disregarding the order of the facts.

Citation

@misc{li2023factual,
  title={FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2305.17497}
}

• arXiv: 2305.17497