Craft Eval

craft-eval

CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation — Ingo Ziegler et al. (arXiv:2409.02098, 2024)

What this evaluates

Evaluates the ability of instruction-tuned LLMs to perform domain-specific multiple-choice question answering and text generation tasks. It measures how well models fine-tuned on synthetic, corpus-retrieved data generalize to held-out human-annotated benchmarks in biology, medicine, commonsense, recipe generation, and summarization.

Datasets

• ScienceQA (BioQA) — total 397; splits: test (397)
• MedMCQA (MedQA) — total 4183; splits: validation (4183)
• CommonsenseQA 2.0 (CSQA) — total 2541; splits: validation (2541)
• RecipeNLG (RecipeGen) — total 1000; splits: test (1000)
• CNN-DailyMail (Summarization) — total 1000; splits: test (1000)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted answer labels, computed by assessing log-probabilities of vocabulary tokens corresponding to options A–E and selecting the argmax via greedy decoding without temperature scaling.
• win rate — range: [0, 1]
  • Proportion of pairwise LLM-judge comparisons where the model's generated output is preferred over the reference, following the Alpaca-Eval benchmark protocol using Llama 3 70B as the annotator.

Input / output format

Input: Multiple-choice questions with answer options (A–E) for QA tasks; raw text excerpts or recipe prompts for generation tasks.

Output: Single correct answer letter label for QA; structured text (ingredients and steps for recipes, or concise summaries) for generation tasks.

Scoring recipe

def score_qa(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

def score_generation(predictions, references):
    wins, losses, ties = 0, 0, 0
    for pred, ref in zip(predictions, references):
        verdict = llm_judge.compare(pred, ref)
        if verdict == 'win': wins += 1
        elif verdict == 'loss': losses += 1
        else: ties += 1
    return wins / (wins + losses + ties)

Common pitfalls

• Avoid ROUGE/METEOR for generation evaluation as the paper explicitly notes their unreliability due to n-gram overlap limitations and reference quality issues.
• Use MMLU-style log-probability accuracy for multiple-choice QA rather than simple greedy token matching or temperature-scaled decoding.
• Do not confuse the synthetic training dataset sizes (XS/S/M/L/XL) with the fixed human-annotated evaluation splits.

Evidence (verbatim from paper)

QA Tasks: We evaluate multiple-choice QA tasks using accuracy, following MMLU’s approach of assessing logarithmic probabilities for vocabulary tokens corresponding to answer labels. We perform greedy decoding without temperature scaling across answer choices ranging from A-B to A-E.

Citation

@misc{ziegler2024craft,
  title={CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation},
  author={Ingo Ziegler et al.},
  year={2024},
  note={arXiv:2409.02098}
}

• arXiv: 2409.02098