qhjqhj00 fife-eval
Financial Instruction Following Evaluation (FIFE) — Matlin et al. (2025) (arXiv:2512.08965, 2025)
What this evaluates
Evaluates large language models' ability to follow complex financial instructions, with a strong emphasis on precise adherence to formatting, structural constraints, and conditional styling requirements. It probes whether models can maintain procedural compliance rather than just semantic correctness.
Datasets
- FIFE — total ?; splits: test (-1)
Metrics
Strict compliance(primary) — range: percent- Percentage of instructions followed with exact adherence to all requested formatting, structural, and styling constraints. Any deviation (e.g., wrong HTML tags, missing table headers) results in a penalty.
Loose compliance— range: percent- Percentage of instructions where the model's response is semantically correct and fulfills the core instruction, even if minor formatting or structural deviations are present.
Input / output format
Input: Financial instruction prompts specifying complex formatting, structural requirements, and conditional styling rules (e.g., specific table headers, conditional bolding, exact HTML tag usage).
Output: Model-generated text that must strictly or loosely adhere to the formatting and structural constraints specified in the prompt.
Scoring recipe
def evaluate(predictions, gold):
strict_correct = 0
loose_correct = 0
for pred, gold in zip(predictions, gold):
# Strict: exact match on all formatting/structural constraints
if matches_exact_format_and_structure(pred, gold):
strict_correct += 1
# Loose: semantic correctness allowed despite minor formatting flaws
if matches_semantic_intent(pred, gold):
loose_correct += 1
strict_score = (strict_correct / len(predictions)) * 100
loose_score = (loose_correct / len(predictions)) * 100
return strict_score, loose_score
Common pitfalls
- Models often produce semantically correct responses but fail strict scoring due to minor formatting deviations (e.g., using an incorrect HTML tag instead of the requested one).
- Reasoning-tuned or conversational models do not necessarily outperform general-purpose models on strict instruction following, as the benchmark penalizes procedural flaws over factual errors.
- Evaluators may mistakenly reward semantic accuracy under the strict metric, ignoring the benchmark's explicit focus on formatting and structural compliance.
Evidence (verbatim from paper)
Our findings reveal a clear hierarchy in instruction-following capabilities, with open-weight models leading the FIFE benchmark. As shown in 1, the top-performing open-weight system (Llama-4Maverick 18B) achieved a compliance of (76.1 strict / 79.5 loose), significantly outperforming the leading proprietary model (65.9 strict / 70.5 loose).
Citation
@misc{matlin2025fife,
title={Financial Instruction Following Evaluation (FIFE)},
author={Matlin et al. (2025)},
year={2025},
note={arXiv:2512.08965}
}
- arXiv: 2512.08965