flask-eval
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets — Ye et al. (2023) (arXiv:2307.10928, 2023)
What this evaluates
Evaluates LLMs on fine-grained alignment capabilities by decomposing instruction-following performance into 12 sub-skills across four domains (Logical Thinking, Background Knowledge, Problem Handling, User Alignment). It measures how well models adhere to specific quality criteria like factuality, logical robustness, and harmlessness on a per-instance basis.
Datasets
- FLASK — total ?; splits: test (-1), hard (-1)
Metrics
FLASK skill score (primary) — range: [1, 5]
- Average of per-instance scores across 12 alignment sub-skills (Logical Robustness, Logical Correctness, Logical Efficiency, Factuality, Commonsense, Comprehension, Insightfulness, Completeness, Metacognition, Readability, Conciseness, Harmlessness). Each skill is rated on a 1–5 scale by GPT-4 acting as the evaluator, then averaged across instances.
Input / output format
Input: Instruction-based text prompt requiring a model response.
Output: Model-generated text response.
Scoring recipe
def compute_flask_score(dataset, model, evaluator='GPT-4'):
skill_scores = {skill: [] for skill in SKILLS}
for instance in dataset:
response = model.generate(instance.prompt)
for skill in instance.skills:
score = evaluator.score(response, rubric=skill) # 1-5 scale
skill_scores[skill].append(score)
return {skill: mean(scores) for skill, scores in skill_scores.items()}
Common pitfalls
- GPT-4 as evaluator may exhibit self-preference or style bias, inflating scores for models with similar output patterns.
- Scores on FLASK-HARD can drop by up to 50% compared to the base set, making difficulty calibration critical.
- Averaging across 12 sub-skills can mask severe weaknesses in specific alignment dimensions like Logical Robustness.
Evidence (verbatim from paper)
We use GPT-4 as the evaluator (Eval LM) for model-based evaluation. Table 9: Comparison of open-source and proprietary models on the whole FLASK evaluation set. The best performance is shown in bold.
Citation
@misc{ye2023flask,
title={FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets},
author={Ye et al. (2023)},
year={2023},
note={arXiv:2307.10928}
}
1---2name: flask-eval3description: Evaluates LLMs on fine-grained alignment capabilities by decomposing instruction-following performance into 12 sub-skills across four domains (Logical Thinking, Background Knowledge, Problem Handling, User Alignment). It measures how well models adhere to specific quality criteria like factuality, logical robustness, and harmlessness on a per-instance basis. Use when the user wants to benchmark on FLASK, or asks about evaluating this task. Reports FLASK skill score.4metadata:5 source: https://skillmd.com/skills/qhjqhj00/flask-eval6 skill_kind: dataset_eval7 source_arxiv: 2307.109288 bibtex_key: ye2023flask9 confidence: high10---1112# flask-eval1314> FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets — Ye et al. (2023) (arXiv:2307.10928, 2023)1516## What this evaluates1718Evaluates LLMs on fine-grained alignment capabilities by decomposing instruction-following performance into 12 sub-skills across four domains (Logical Thinking, Background Knowledge, Problem Handling, User Alignment). It measures how well models adhere to specific quality criteria like factuality, logical robustness, and harmlessness on a per-instance basis.1920## Datasets2122- **FLASK** — total ?; splits: test (-1), hard (-1)2324## Metrics2526- `FLASK skill score` **(primary)** — range: [1, 5]27 - Average of per-instance scores across 12 alignment sub-skills (Logical Robustness, Logical Correctness, Logical Efficiency, Factuality, Commonsense, Comprehension, Insightfulness, Completeness, Metacognition, Readability, Conciseness, Harmlessness). Each skill is rated on a 1–5 scale by GPT-4 acting as the evaluator, then averaged across instances.2829## Input / output format3031**Input**: Instruction-based text prompt requiring a model response.3233**Output**: Model-generated text response.3435## Scoring recipe3637```python38def compute_flask_score(dataset, model, evaluator='GPT-4'):39 skill_scores = {skill: [] for skill in SKILLS}40 for instance in dataset:41 response = model.generate(instance.prompt)42 for skill in instance.skills:43 score = evaluator.score(response, rubric=skill) # 1-5 scale44 skill_scores[skill].append(score)45 return {skill: mean(scores) for skill, scores in skill_scores.items()}46```4748## Common pitfalls4950- GPT-4 as evaluator may exhibit self-preference or style bias, inflating scores for models with similar output patterns.51- Scores on FLASK-HARD can drop by up to 50% compared to the base set, making difficulty calibration critical.52- Averaging across 12 sub-skills can mask severe weaknesses in specific alignment dimensions like Logical Robustness.5354## Evidence (verbatim from paper)5556> We use GPT-4 as the evaluator (Eval LM) for model-based evaluation. Table 9: Comparison of open-source and proprietary models on the whole FLASK evaluation set. The best performance is shown in bold.5758## Citation5960```bibtex61@misc{ye2023flask,62 title={FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets},63 author={Ye et al. (2023)},64 year={2023},65 note={arXiv:2307.10928}66}67```6869- arXiv: 2307.10928