Scene Graph Modification Eval

scene-graph-modification-eval

Scene Graph Modification as Incremental Structure Expanding — Hu et al. (2022) (arXiv:2209.09093, 2022)

What this evaluates

Evaluates a model's ability to modify a source scene graph into a target scene graph conditioned on a natural language query. It probes incremental structure expansion, joint node-edge prediction, and the preservation of unmodified graph components during editing.

Datasets

• User Generated — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/THU-BPM/SGM
• MSCOCO — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/THU-BPM/SGM
• GCC — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/THU-BPM/SGM
• RSICD — total 10000; splits: train (8000), dev (1000), test (1000); repo https://github.com/THU-BPM/SGM

Metrics

• Node F1 — range: percent
  • Standard F1 score computed over predicted versus ground-truth node types/labels across the entire graph. Precision and recall are calculated based on exact node type matching.
• Edge F1 — range: percent
  • Standard F1 score computed over predicted versus ground-truth edge relations. Precision and recall are calculated based on exact relation type matching between node pairs.
• Graph-level accuracy (primary) — range: percent
  • Exact string match between the entire predicted scene graph and the target scene graph. The prediction is counted as correct only if every node and edge exactly matches the ground truth.

Input / output format

Input: A natural language query and a source scene graph.

Output: A target scene graph consisting of predicted nodes and edges.

Scoring recipe

def compute_metrics(pred_graph, gold_graph):
    pred_nodes = set(pred_graph.nodes)
    gold_nodes = set(gold_graph.nodes)
    node_f1 = f1_score(gold_nodes, pred_nodes)
    pred_edges = set(pred_graph.edges)
    gold_edges = set(gold_graph.edges)
    edge_f1 = f1_score(gold_edges, pred_edges)
    graph_acc = 1.0 if pred_graph == gold_graph else 0.0
    return node_f1, edge_f1, graph_acc

Common pitfalls

• Graph-level accuracy requires a strict exact structural match, not just semantic equivalence or partial overlap.
• Edge prediction is significantly harder than node prediction, especially on datasets with high data sparsity or large graph sizes.
• Evaluating on full graph regeneration baselines versus incremental modification baselines yields different results; the protocol assumes modification-aware evaluation.

Evidence (verbatim from paper)

Following Weber et al. (2021), we use three automatic metrics for the evaluation, including node-level and edge-level F1 score, and graph-level accuracy. Graph-level accuracy is computed based on exact string match, which requires the generated scene graph to be identical to the target scene graph for a correct prediction.

Citation

@misc{hu2022scenegraphmodification,
  title={Scene Graph Modification as Incremental Structure Expanding},
  author={Hu et al. (2022)},
  year={2022},
  note={arXiv:2209.09093}
}

• arXiv: 2209.09093