Flask Eval

flask-eval

FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets — Ye et al. (2023) (arXiv:2307.10928, 2023)

What this evaluates

Evaluates LLMs on fine-grained alignment capabilities by decomposing instruction-following performance into 12 sub-skills across four domains (Logical Thinking, Background Knowledge, Problem Handling, User Alignment). It measures how well models adhere to specific quality criteria like factuality, logical robustness, and harmlessness on a per-instance basis.

Datasets

• FLASK — total ?; splits: test (-1), hard (-1)

Metrics

• FLASK skill score (primary) — range: [1, 5]
  • Average of per-instance scores across 12 alignment sub-skills (Logical Robustness, Logical Correctness, Logical Efficiency, Factuality, Commonsense, Comprehension, Insightfulness, Completeness, Metacognition, Readability, Conciseness, Harmlessness). Each skill is rated on a 1–5 scale by GPT-4 acting as the evaluator, then averaged across instances.

Input / output format

Input: Instruction-based text prompt requiring a model response.

Output: Model-generated text response.

Scoring recipe

def compute_flask_score(dataset, model, evaluator='GPT-4'):
    skill_scores = {skill: [] for skill in SKILLS}
    for instance in dataset:
        response = model.generate(instance.prompt)
        for skill in instance.skills:
            score = evaluator.score(response, rubric=skill)  # 1-5 scale
            skill_scores[skill].append(score)
    return {skill: mean(scores) for skill, scores in skill_scores.items()}

Common pitfalls

• GPT-4 as evaluator may exhibit self-preference or style bias, inflating scores for models with similar output patterns.
• Scores on FLASK-HARD can drop by up to 50% compared to the base set, making difficulty calibration critical.
• Averaging across 12 sub-skills can mask severe weaknesses in specific alignment dimensions like Logical Robustness.

Evidence (verbatim from paper)

We use GPT-4 as the evaluator (Eval LM) for model-based evaluation. Table 9: Comparison of open-source and proprietary models on the whole FLASK evaluation set. The best performance is shown in bold.

Citation

@misc{ye2023flask,
  title={FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets},
  author={Ye et al. (2023)},
  year={2023},
  note={arXiv:2307.10928}
}

• arXiv: 2307.10928