# Possible Stories Ifsm Eval

> Assesses whether large language models can generate story endings that align with free-form instructions provided alongside a narrative context. It measures both instruction-following accuracy and the model's ability to produce distinct endings for different instructions. Use when the user wants to benchmark on Possible Stories, or asks about evaluating this task. Reports IFSM.

- Skill: `qhjqhj00/possible-stories-ifsm-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/possible-stories-ifsm-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/possible-stories-ifsm-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/possible-stories-ifsm-eval

---


# possible-stories-ifsm-eval

> Evaluation of Instruction-Following Ability for Large Language Models on Story-Ending Generation — Hida et al. (2024) (arXiv:2406.16356, 2024)

## What this evaluates

Assesses whether large language models can generate story endings that align with free-form instructions provided alongside a narrative context. It measures both instruction-following accuracy and the model's ability to produce distinct endings for different instructions.

## Datasets

- **Possible Stories** — total ?; splits: test (-1); repo https://github.com/nii-cl/possible-stories

## Metrics

- `IFSM` **(primary)** — range: [0, 1]
  - Instruction Following Score from MRC. A binary classification score from a Machine Reading Comprehension model that predicts whether a generated story ending follows the given instruction. Calculated as the proportion of instances predicted as 'Follow'.
- `Dissimilarity` — range: [0, 1]
  - Measures controllability by computing the average semantic distance between ending pairs generated from the same context but different instructions. Formula: 1 - cos(LaBSE(e_i), LaBSE(e_j)).

## Input / output format

**Input**: Narrative context and a free-form instruction/question from the Possible Stories dataset.

**Output**: A free-form story ending generated by the LLM.

## Scoring recipe

```python
# IFSM (Instruction Following Score from MRC)
ifsm_preds = [mrc_model.predict(context, instruction, ending) for ending in generated_endings]
ifsm_score = sum(1 for p in ifsm_preds if p == "Follow") / len(ifsm_preds)

# Dissimilarity (Controllability)
dissim_scores = []
for e_i, e_j in ending_pairs_from_same_context:
    sim = cosine_similarity(LaBSE(e_i), LaBSE(e_j))
    dissim_scores.append(1 - sim)
dissimilarity_score = sum(dissim_scores) / len(dissim_scores)
```

## Common pitfalls

- IFSM relies on an external MRC model for scoring rather than exact match or human judgment, so results depend heavily on the MRC model's calibration and prompt format.
- Dissimilarity is computed over pairs of endings for the same context; the number of pairs and sampling strategy significantly affect the average score, making cross-model comparisons sensitive to pair generation protocols.

## Evidence (verbatim from paper)

> We introduced a controllability metric (Dissimilarity) other than IFSM because LLMs should generate different endings based on different instructions and IFSM can’t evaluate it. We used LaBSE*(Feng et al., [2022])* to measure the semantic dissimilarity between ending pairs ($e_{i}$, $e_{j}$) from the same context with different instructions, formulated as 1 - $cos(	ext{LaBSE}(e_{i}),	ext{LaBSE}(e_{j}))$777We’ve confirmed using another sentence similarity model..

## Citation

```bibtex
@misc{hida2024storyending,
  title={Evaluation of Instruction-Following Ability for Large Language Models on Story-Ending Generation},
  author={Hida et al. (2024)},
  year={2024},
  note={arXiv:2406.16356}
}
```

- arXiv: 2406.16356