Fluke Robustness Eval

fluke-robustness-eval

FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation — Otmakhova et al. (2025) (arXiv:2504.17311, 2025)

What this evaluates

Evaluates how well NLP models maintain performance when subjected to minimal, linguistically-grounded perturbations (e.g., syntactic voice changes, negation, style shifts, geographical/temporal biases) across classification and generation tasks. It probes model brittleness to covariate shifts introduced by natural language modifications rather than adversarial noise.

Datasets

• KnowRef — total ?; splits: (unstated)
• Few-NERD — total ?; splits: (unstated)
• GSM8K — total ?; splits: (unstated)
• IFEval — total ?; splits: (unstated)

Metrics

• Unrobustness (U, %) (primary) — range: percent
  • Percentage performance degradation on modified instances relative to baseline. Computed as the drop in accuracy (for classification tasks) or exact-match correctness (for generation tasks) after applying a linguistic modification, averaged across instances for each modification type.

Input / output format

Input: Task-specific input (sentence for NER/Coreference, math word problem for GSM8K, instruction prompt for IFEval) with a targeted linguistic modification applied (e.g., active-to-passive voice, negation insertion, style/dialect shift, or bias injection).

Output: NER: entity type labels; Coreference: index of correct candidate; GSM8K: final computed numerical answer; IFEval: generated text satisfying verifiable constraints.

Scoring recipe

def compute_unrobustness(baseline_acc, modified_acc):
    if baseline_acc == 0:
        return 0.0
    return ((baseline_acc - modified_acc) / baseline_acc) * 100

# Applied per modification type across the dataset.
# Final U(%) is the average across instances for that modification.

Common pitfalls

• Modifications can inadvertently change the ground truth or make tasks unsolvable; the authors manually verify label validity post-modification before reporting results.
• Unrobustness is reported per linguistic category (e.g., Syntax, Semantics, Bias), not just a single aggregate score; readers must inspect specific modification types to understand brittleness patterns.
• Generation tasks (GSM8K, IFEval) are evaluated only on LLMs, not PLMs, due to architectural constraints and task nature, which limits direct cross-architecture comparison for those tasks.

Evidence (verbatim from paper)

Interestingly, unrobustness results for NER (see Table 2) and Coreference (Table 3) show different, almost orthogonal trends in terms of the tests the models were brittle to. Table 2: NER: Unrobustness (U, %) by model and modification.

Citation

@misc{otmakhova2025fluke,
  title={FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation},
  author={Otmakhova et al. (2025)},
  year={2025},
  note={arXiv:2504.17311}
}

• arXiv: 2504.17311