Stage QA Eval

stage-qa-eval

STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays — Qiuyu Tian et al. (2026) (arXiv:2601.08510, 2026)

What this evaluates

Tests retrieval-augmented question answering over long screenplay contexts. It probes the model's ability to locate relevant information across chunked text and synthesize accurate answers under different retrieval architectures.

Datasets

• STAGE-QA — total 4162; splits: test (4162)

Metrics

• Question Correctness (primary) — range: [0, 1]
  • Binary correctness per question. Answers are generated five times per question; the question is marked correct if any generation is judged correct by GPT-4o.

Input / output format

Input: A curated screenplay question, optionally accompanied by retrieved context chunks (300-token, 200-token child/400-token parent, or GraphRAG summaries) depending on the retrieval setting.

Output: Free-form natural language answer to the question.

Scoring recipe

def score_qa(question, retrieved_context):
    answers = [generate_answer(question, retrieved_context, temp=0.2) for _ in range(5)]
    correct = any(llm_judge_gpt4o(f'Q: {question}\nA: {ans}\nIs this correct?') for ans in answers)
    return {'question_correctness': 1.0 if correct else 0.0}

Common pitfalls

• Correctness is judged by GPT-4o, not exact string matching, so paraphrased but correct answers are accepted.
• A question is marked correct if any of the 5 generations is judged correct, which inflates correctness compared to single-generation evaluation.
• Retrieval settings vary (native RAG, hybrid child-parent, GraphRAG), so context length and chunking strategy affect performance.

Evidence (verbatim from paper)

For each question, answers are generated five times; a question is marked correct if any generation is judged correct.

Citation

@misc{tian2026stage,
  title={STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays},
  author={Qiuyu Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.08510}
}

• arXiv: 2601.08510