Stage Es Eval

stage-es-eval

STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays — Qiuyu Tian et al. (2026) (arXiv:2601.08510, 2026)

What this evaluates

Assesses an LLM's capacity to abstract scene-level events into concise, free-form descriptions without schema constraints. It evaluates whether the generated events form a coherent, non-redundant structure and remain factually grounded in the screenplay text.

Datasets

• STAGE-ES — total 13750; splits: test (13750)

Metrics

• Event-Structure Consistency (primary) — range: [0, 1]
  • LLM-judged score (via GPT-4o) measuring whether predicted events form a coherent, non-redundant abstraction of the scene.
• Factual Supportfulness — range: [0, 1]
  • LLM-judged score measuring whether each predicted event is grounded in explicit screenplay evidence and avoids hallucinations.

Input / output format

Input: Full text of a single movie scene.

Output: Concise, free-form event description(s) with no constraints on schema, ordering, or event count.

Scoring recipe

def score_es(pred_events, gold_scene_text):
    consistency_score = llm_judge_gpt4o(
        prompt=f'Scene: {gold_scene_text}\nPredicted events: {pred_events}\nRate event-structure consistency (0-1).'
    )
    support_score = llm_judge_gpt4o(
        prompt=f'Scene: {gold_scene_text}\nPredicted events: {pred_events}\nRate factual supportfulness (0-1).'
    )
    return {'event_structure_consistency': consistency_score, 'factual_supportfulness': support_score}

Common pitfalls

• Evaluation uses LLM-as-a-judge (GPT-4o) rather than exact string overlap, so scores depend on the judge's prompting and temperature.
• The metric emphasizes event-level abstraction and structural coherence rather than sentence-level fact decomposition, unlike standard FActScore.
• Models are not constrained to a fixed schema or event count, making automated exact-match evaluation impossible.

Evidence (verbatim from paper)

Evaluation follows the Event-Structure Consistency framework along two dimensions: event-structure consistency, which measures whether predicted events form a coherent, non-redundant abstraction of the scene; and factual supportfulness, which measures whether each event is grounded in explicit screenplay evidence and avoids unsupported hallucinations.

Citation

@misc{tian2026stage,
  title={STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays},
  author={Qiuyu Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.08510}
}

• arXiv: 2601.08510