Possible Stories Ifsm Eval

possible-stories-ifsm-eval

Evaluation of Instruction-Following Ability for Large Language Models on Story-Ending Generation — Hida et al. (2024) (arXiv:2406.16356, 2024)

What this evaluates

Assesses whether large language models can generate story endings that align with free-form instructions provided alongside a narrative context. It measures both instruction-following accuracy and the model's ability to produce distinct endings for different instructions.

Datasets

• Possible Stories — total ?; splits: test (-1); repo https://github.com/nii-cl/possible-stories

Metrics

• IFSM (primary) — range: [0, 1]
  • Instruction Following Score from MRC. A binary classification score from a Machine Reading Comprehension model that predicts whether a generated story ending follows the given instruction. Calculated as the proportion of instances predicted as 'Follow'.
• Dissimilarity — range: [0, 1]
  • Measures controllability by computing the average semantic distance between ending pairs generated from the same context but different instructions. Formula: 1 - cos(LaBSE(e_i), LaBSE(e_j)).

Input / output format

Input: Narrative context and a free-form instruction/question from the Possible Stories dataset.

Output: A free-form story ending generated by the LLM.

Scoring recipe

# IFSM (Instruction Following Score from MRC)
ifsm_preds = [mrc_model.predict(context, instruction, ending) for ending in generated_endings]
ifsm_score = sum(1 for p in ifsm_preds if p == "Follow") / len(ifsm_preds)

# Dissimilarity (Controllability)
dissim_scores = []
for e_i, e_j in ending_pairs_from_same_context:
    sim = cosine_similarity(LaBSE(e_i), LaBSE(e_j))
    dissim_scores.append(1 - sim)
dissimilarity_score = sum(dissim_scores) / len(dissim_scores)

Common pitfalls

• IFSM relies on an external MRC model for scoring rather than exact match or human judgment, so results depend heavily on the MRC model's calibration and prompt format.
• Dissimilarity is computed over pairs of endings for the same context; the number of pairs and sampling strategy significantly affect the average score, making cross-model comparisons sensitive to pair generation protocols.

Evidence (verbatim from paper)

We introduced a controllability metric (Dissimilarity) other than IFSM because LLMs should generate different endings based on different instructions and IFSM can’t evaluate it. We used LaBSE*(Feng et al., [2022])* to measure the semantic dissimilarity between ending pairs ($e_{i}$, $e_{j}$) from the same context with different instructions, formulated as 1 - $cos( ext{LaBSE}(e_{i}), ext{LaBSE}(e_{j}))$777We’ve confirmed using another sentence similarity model..

Citation

@misc{hida2024storyending,
  title={Evaluation of Instruction-Following Ability for Large Language Models on Story-Ending Generation},
  author={Hida et al. (2024)},
  year={2024},
  note={arXiv:2406.16356}
}

• arXiv: 2406.16356