Stage Kg Eval

stage-kg-eval

STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays — Qiuyu Tian et al. (2026) (arXiv:2601.08510, 2026)

What this evaluates

Evaluates an LLM's ability to extract and structure narrative knowledge from full-length movie screenplays into canonical graphs. It probes entity and relation recognition while filtering out event-centric noise to focus on stable world-building elements.

Datasets

• STAGE-KG — total 150; splits: test (150)

Metrics

• Entity F1 (primary) — range: [0, 1]
  • Standard F1 score computed over entity matches after normalization and embedding-based similarity matching. Precision and recall are calculated against canonical graphs restricted to reference entities with degree ≥ 2.
• Relation F1 — range: [0, 1]
  • Standard F1 score for relation extraction, computed under the same degree ≥ 2 and reference-only constraints as entities.

Input / output format

Input: Full screenplay text segmented into 600-token chunks (for zero-shot/GraphRAG baselines) or the complete script, depending on the baseline.

Output: Extracted entity-relation graph (nodes and edges) representing the movie's narrative world.

Scoring recipe

def score_kg(pred_graph, gold_graph):
    gold_filtered = filter_degree_and_type(gold_graph, min_degree=2, exclude_event=True)
    pred_filtered = filter_degree_and_type(pred_graph, min_degree=2, exclude_event=True)
    matched_entities = match_entities(pred_filtered.entities, gold_filtered.entities)
    matched_relations = match_relations(pred_filtered.relations, gold_filtered.relations, matched_entities)
    prec_e = len(matched_entities) / max(len(pred_filtered.entities), 1)
    rec_e = len(matched_entities) / max(len(gold_filtered.entities), 1)
    f1_e = 2 * prec_e * rec_e / (prec_e + rec_e)
    prec_r = len(matched_relations) / max(len(pred_filtered.relations), 1)
    rec_r = len(matched_relations) / max(len(gold_filtered.relations), 1)
    f1_r = 2 * prec_r * rec_r / (prec_r + rec_r)
    return {'entity_f1': f1_e, 'relation_f1': f1_r}

Common pitfalls

• Event nodes and event-centric relations are explicitly excluded from scoring, but models often extract them anyway.
• Only reference entities and relations with a degree ≥ 2 are scored; low-degree or event-specific elements are ignored in the ground truth.
• Entity matching relies on both normalization rules and embedding similarity, not exact string matching.

Evidence (verbatim from paper)

Evaluation reports entity and relation precision/recall/F1 against canonical graphs, restricted to narratively salient structure: event nodes and event-centric relations are excluded, and only reference entities and relations with degree ≥ 2 are scored.

Citation

@misc{tian2026stage,
  title={STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays},
  author={Qiuyu Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.08510}
}

• arXiv: 2601.08510