Stage Icrp Eval

stage-icrp-eval

STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays — Qiuyu Tian et al. (2026) (arXiv:2601.08510, 2026)

What this evaluates

Evaluates an LLM's ability to role-play as a specific movie character using memory-grounded agent frameworks. It probes consistency with the character's persona, speaking style, and narrative facts across interactive dialogues.

Datasets

• STAGE-ICRP — total 1505; splits: test (1505)

Metrics

• Persona Consistency (primary) — range: [0, 1]
  • LLM-judged score (via GPT-4o) measuring whether the model's responses align with the character's established persona.
• Speaking Style Consistency — range: [0, 1]
  • LLM-judged score measuring whether the model maintains the character's unique linguistic patterns and tone.
• Narrative Faithfulness — range: [0, 1]
  • LLM-judged score measuring whether the model's responses remain consistent with verified character facts and plot events.

Input / output format

Input: A persona specification derived from screenplay evidence, optionally augmented with narrative episodic memory summaries and explicit character facts, plus an audience-facing interaction question.

Output: First-person natural language response from the perspective of the specified character.

Scoring recipe

def score_icrp(pred_response, persona_spec, gold_facts):
    persona_score = llm_judge_gpt4o(f'Persona: {persona_spec}\nResponse: {pred_response}\nRate persona consistency.')
    style_score = llm_judge_gpt4o(f'Persona: {persona_spec}\nResponse: {pred_response}\nRate speaking style consistency.')
    faith_score = llm_judge_gpt4o(f'Facts: {gold_facts}\nResponse: {pred_response}\nRate narrative faithfulness.')
    return {'persona_consistency': persona_score, 'speaking_style_consistency': style_score, 'narrative_faithfulness': faith_score}

Common pitfalls

• Evaluation relies entirely on GPT-4o as an LLM judge, so scores may reflect the judge's biases rather than ground-truth character alignment.
• Models are tested with three different memory configurations (Prompt-only, +Episodic Memory, +Episodic Memory + Facts), so performance heavily depends on the memory retrieval setup.
• First-person role-playing responses are inherently subjective, making deterministic scoring impossible without LLM judges.

Evidence (verbatim from paper)

Evaluation is conducted along three dimensions: persona consistency, speaking style consistency, and narrative faithfulness. Full prompts and judging criteria are provided in Appendix [D].

Citation

@misc{tian2026stage,
  title={STAGE: A Benchmark for Knowledge Graph Construction, Question Answering, and In-Script Role-Playing over Movie Screenplays},
  author={Qiuyu Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.08510}
}

• arXiv: 2601.08510