Event Driven Storytelling Eval

event-driven-storytelling-eval

Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scene — Lim et al. (2025) (arXiv:2507.19232, 2025)

What this evaluates

Evaluates an LLM's ability to perform spatial and contextual reasoning for multi-agent planning in 3D scenes. It tests object arrangement, regional context alignment, and scene state tracking to generate plausible character actions and positions.

Datasets

• Event-Driven Storytelling Benchmark — total 40; splits: test (40)

Metrics

• success rate (primary) — range: [0, 1]
  • Proportion of test case runs that pass all three conditions: a plan is generated, the action label is valid and matches expectations, and the character's position falls within the expected area or coordinate.
• execution rate — range: [0, 1]
  • Proportion of test case runs that complete without runtime errors, such as the LLM referencing nonexistent objects or generating syntactically invalid responses.

Input / output format

Input: Scene description, test scenario (T), and mock planning history (H) simulating prior agent actions.

Output: Next plan τ_g = (a_g, p_g or ilde{p}_g), where a_g is an action label, and p_g is a coordinate or ilde{p}_g is a semantic area.

Scoring recipe

def evaluate(generated_plan, expected_plan, available_actions, expected_area, expected_coords):
    if not generated_plan:
        return False
    action = generated_plan.action
    if action not in available_actions or action not in expected_plan.actions:
        return False
    if expected_plan.is_area_level:
        return generated_plan.area in expected_area
    else:
        return generated_plan.coord in expected_coords

Common pitfalls

• Runtime errors from LLM hallucinations or syntax issues automatically mark a run as failed, regardless of reasoning quality.
• Test cases are split into area-level and coordinate-level outputs; using the wrong comparison logic causes false negatives.
• Mock planning history must accurately simulate prior steps; incorrect context breaks state-reasoning evaluations.

Evidence (verbatim from paper)

Our benchmark employs two evaluation metrics: success rate and execution rate. The success rate measures the proportion of passed runs among all test case runs. On the other hand, the execution rate measures the proportion of runs that are executed without any runtime errors, which are included in the parenthesis next to the success rate in tables.

Citation

@misc{lim2025eventdrivenstorytelling,
  title={Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scene},
  author={Lim et al. (2025)},
  year={2025},
  note={arXiv:2507.19232}
}

• arXiv: 2507.19232