Longform C Eval

longform-c-eval

LongForm: Effective Instruction Tuning with Reverse Instructions — Köksal et al. (2023) (arXiv:2304.08460, 2023)

What this evaluates

Evaluates instruction-following long-form text generation capabilities of LLMs across diverse in-domain and out-of-domain tasks, including news summarization, recipe and story generation, long-form QA, and multilingual generation, while also measuring general language understanding via MMLU.

Datasets

• LongForm-C — total ?; splits: test (-1)
• Writing Prompts — total 250; splits: test (250)
• ELI5 — total 250; splits: test (250)
• Recipe Generation — total 250; splits: test (250)
• MMLU — total ?; splits: test (-1)
• MLSUM — total 400; splits: test (400)

Metrics

• METEOR (primary) — range: [0, 1] (reported as 0–100 in tables)
  • Computes unigram precision and recall, applies a harmonic mean (F-mean), and penalizes fragmentation based on the longest common subsequence between reference and hypothesis.
• BLEU — range: [0, 1] (reported as 0–100)
  • Measures n-gram precision between reference and hypothesis with a brevity penalty for shorter outputs.
• ROUGE — range: [0, 1] (reported as 0–100)
  • Recall-oriented metric measuring overlap of n-grams, longest common subsequence, or summary words between reference and hypothesis.
• self-BLEU — range: [0, 1] (reported as 0–100)
  • Measures diversity within a set of generated hypotheses by computing BLEU scores between each pair of outputs.

Input / output format

Input: Natural language instruction specifying the task, optionally with context/ingredients/title. For autoregressive models, an [EOI] token is appended between the instruction and the generation start to separate instruction from output.

Output: Generated long-form text response to the instruction.

Scoring recipe

def compute_metric(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        scores.append(meteor_score([ref], pred))
    return sum(scores) / len(scores)

Common pitfalls

• Relying on BLEU/ROUGE for long-form evaluation, as the paper explicitly notes they have limited correlation with human judgment for long texts.
• Assuming out-of-domain generalization datasets (WP, ELI5, Recipe) were part of the training data; they are explicitly held out to test generalization.
• Ignoring the [EOI] token requirement for autoregressive models, which is critical for correctly separating instruction from generation during inference.

Evidence (verbatim from paper)

As current metrics in text generation have limited capabilities in evaluating long text generation Celikyilmaz et al. ([2020]), we choose METEOR Banerjee and Lavie ([2005]) as our main metric as it exhibits higher human correlation Sharma et al. ([2017]); Chen et al. ([2022]). We evaluate them on the test set of LongForm-C... For generation, we perform nucleus sampling with p=0.9 for all LMs.

Citation

@misc{koksal2023longform,
  title={LongForm: Effective Instruction Tuning with Reverse Instructions},
  author={Köksal et al. (2023)},
  year={2023},
  note={arXiv:2304.08460}
}

• arXiv: 2304.08460