qhjqhj00 instruction-adherence-eval
When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs — Young et al. (2025) (arXiv:2510.18892, 2025)
What this evaluates
This benchmark probes instruction-following capabilities by testing models on 20 carefully designed prompts that enforce format compliance, content constraints, logical sequencing, and multi-step execution. It measures whether models can adhere to verifiable, unambiguous constraints rather than relying on superficial pattern matching or memorized benchmark performance.
Datasets
- Instruction Adherence Diagnostic Prompts — total 20; splits: test (20)
Metrics
binary_pass_fail(primary) — range: percent- Binary pass/fail determination based on programmatic verification of adherence to specified constraints. A response is marked as pass if it satisfies strict matching criteria or acceptable semantic variations (e.g., whitespace, quote styles, markdown wrappers).
response_time— range: seconds- Wall-clock time in seconds from API request to response completion, capped at a 10-second timeout.
token_usage— range: tokens- Total number of tokens consumed (input + output) for each test execution.
Input / output format
Input: A single diagnostic prompt containing explicit, verifiable constraints (e.g., sequential steps, forbidden characters, JSON formatting requirements).
Output: Free-form text response that must strictly adhere to the prompt's constraints (e.g., valid JSON, specific letter avoidance, ordered steps).
Scoring recipe
def score(prediction, gold_criteria):
# Stage 1: Strict matching
if matches_strict(prediction, gold_criteria):
return 1
# Stage 2: Flexible verification (handles whitespace, quotes, markdown wrappers)
if matches_flexible(prediction, gold_criteria):
return 1
return 0
# Aggregation: pass_rate = (sum(score) / total_tests) * 100
Common pitfalls
- API parameter mismatches (e.g., unsupported temperature or reasoning parameters) can cause false negatives if not dynamically filtered per model metadata.
- Timeouts or rate limits on free tiers may incorrectly flag a model as non-functional rather than slow or restricted.
- Models may appear to follow instructions via superficial pattern matching rather than genuine constraint understanding, leading to overestimation of true instruction-following capability.
Evidence (verbatim from paper)
Primary metrics include binary pass/fail determination based on adherence to specified instructions, response time measurement for performance characterization, and token usage statistics for efficiency analysis.
Citation
@misc{young2025when,
title={When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs},
author={Young et al. (2025)},
year={2025},
note={arXiv:2510.18892}
}
- arXiv: 2510.18892