Sft Generalization Eval

sft-generalization-eval

Debunk the Myth of SFT Generalization — Lin et al. (2025) (arXiv:2510.00237, 2025)

What this evaluates

Probes whether language models trained via supervised fine-tuning (SFT) truly learn reasoning and planning capabilities or merely memorize instruction templates. It tests generalization across unseen action mappings (instruction variations) and increased grid/card complexity (difficulty variations).

Datasets

• Sokoban — total 3981; splits: train (3981), test (-1); repo https://github.com/XiaofengLin7/debunking-sft-generalization
• General Points — total 10000; splits: train (10000), test (-1); repo https://github.com/XiaofengLin7/debunking-sft-generalization

Metrics

• exact-match accuracy (primary) — range: percent
  • Percentage of test instances where the model's predicted action sequence or arithmetic expression exactly matches the gold solution. Computed as (correct predictions / total test instances) * 100.

Input / output format

Input: For Sokoban: grid state encoding, step limit, and action instruction mapping. For General Points: a set of 4 or 5 card values and a target number (default 24).

Output: For Sokoban: a sequence of actions (e.g., 'up, down, left, right' or mapped symbols). For General Points: a valid arithmetic expression using the given cards exactly once that equals the target.

Scoring recipe

def score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred.strip() == gold.strip():
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Models may rely on frozen-prompt artifacts, rigidly adhering to training instruction templates rather than learning the underlying task logic.
• Instruction variations (e.g., numerical/alphabetical mappings) can be confused with difficulty variations, making it hard to isolate true generalization.
• Chain-of-thought supervision traces are generated via rejection sampling from an RL-finetuned model, which may introduce distributional biases not present in human demonstrations.

Evidence (verbatim from paper)

We evaluate generalization on two tasks that expose both instruction variations and difficulty variations, a design partially inspired by Huang et al. [6]. Sokoban is a multi-step puzzle environment requiring long-horizon planning to avoid dead-ends, while General Points is an arithmetic reasoning task.

Citation

@misc{lin2025debunksft,
  title={Debunk the Myth of SFT Generalization},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2510.00237}
}

• arXiv: 2510.00237