Simplestories Diversity Eval

simplestories-diversity-eval

Parameterized Synthetic Text Generation with SimpleStories — Finke et al. (2025) (arXiv:2504.09184, 2025)

What this evaluates

Evaluates the lexical, semantic, and syntactic diversity of a synthetic story dataset compared to a baseline. It measures n-gram distribution, compression ratio, Self-BLEU, n-gram diversity scores, POS template rates, and model-judged semantic variation to assess how well the dataset avoids formulaic phrasing and redundancy.

Datasets

• SimpleStories — total ?; splits: full (-1); repo https://github.com/lennart-finke/simple_stories_generate

Metrics

• compression_ratio (primary) — range: other
  • Measures diversity via document compression relative to original size; higher compression ratios imply more redundancy.
• self_bleu — range: [0, 1]
  • Computes the mean BLEU similarity over every pair of stories in the dataset to capture aggregate text homogenization.
• ngram_diversity_score — range: [0, 1]
  • Ratio of unique n-gram counts to total n-gram counts in a document, capturing repeated sequences beyond single-token diversity.

Input / output format

Input: Random subsamples of story texts from the dataset (e.g., 10% of total or fixed size 1000).

Output: Aggregated metric values (ratios, percentages, or scores) computed over the subsample.

Scoring recipe

def compute_ngram_diversity(stories, n):
    total_ngrams = 0
    unique_ngrams = set()
    for story in stories:
        ngrams = [story[i:i+n] for i in range(len(story)-n+1)]
        total_ngrams += len(ngrams)
        unique_ngrams.update(ngrams)
    return len(unique_ngrams) / total_ngrams if total_ngrams > 0 else 0.0

Common pitfalls

• Parameter counts differ in definition: SimpleStories model sizes include embedding parameters, while TinyStories sizes exclude them, leading to potential unfair comparisons if not normalized.
• Diversity metrics are evaluated on random subsamples (10% or N=1000), which may not capture full dataset distribution or long-tail n-grams.
• Semantic diversity relies on GPT-4o-mini as a judge, introducing potential model-specific biases in simplicity and style scoring.

Evidence (verbatim from paper)

We also measure the compression ratio and Self-BLEU homogenization score, two diversity metrics with low mutual correlation (Shaib et al., [2025]), on a random subsample of size 1000 for each dataset via the diversity package. Computing the compression ratio allows us to measure diversity in terms of document compression relative to original size —higher compression ratios imply more redundancy. Self-homogenization scores capture aggregate similarity by computing the mean similarity over every pair of stories in the dataset.

Citation

@misc{finke2025simplestories,
  title={Parameterized Synthetic Text Generation with SimpleStories},
  author={Finke et al. (2025)},
  year={2025},
  note={arXiv:2504.09184}
}

• arXiv: 2504.09184