Off Policy Sft Eval

off-policy-sft-eval

Patch the Distribution Mismatch: RL Rewriting Agent for Stable Off-Policy SFT — Wang et al. (2026) (arXiv:2602.11220, 2026)

What this evaluates

Evaluates the trade-off between improving downstream mathematical reasoning capabilities and mitigating catastrophic forgetting on general-domain knowledge benchmarks after off-policy supervised fine-tuning. It measures how well a model retains pre-trained general knowledge while learning a new specialized task.

Datasets

• Math500 — total ?; splits: test (-1)
• MinervaMath — total ?; splits: test (-1)
• AMC23 — total ?; splits: test (-1)
• AGIEval-Math — total ?; splits: test (-1)
• IMO-Bench — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• AGIEval — total ?; splits: test (-1)

Metrics

• MathAvg — range: [0, 1]
  • Arithmetic mean of accuracy scores across all five mathematical reasoning benchmarks.
• GeneralAvg — range: [0, 1]
  • Arithmetic mean of accuracy scores across MMLU, MMLU-Pro, and AGIEval (with math-related subsets removed).
• OverallAvg (primary) — range: [0, 1]
  • Average of MathAvg and GeneralAvg: (MathAvg + GeneralAvg) / 2. Used as the headline metric for the gain-forgetting trade-off.
• Math↑ (%) — range: percent
  • Relative improvement in MathAvg over the instruct-tuned base model: (M - M_base) / M_base × 100.
• Gen↓ (%) — range: percent
  • Relative drop in GeneralAvg compared to the instruct-tuned base model: (G_base - G) / G_base × 100.

Input / output format

Input: Instruction prompt for mathematical reasoning or general-domain multiple-choice/factual QA tasks.

Output: Model-generated answer text or selected option label.

Scoring recipe

def compute_metrics(preds, golds, base_preds):
    math_scores = [exact_match(p, g) for p, g in zip(preds['math'], golds['math'])]
    general_scores = [exact_match(p, g) for p, g in zip(preds['gen'], golds['gen'])]
    
    math_avg = mean(math_scores)
    general_avg = mean(general_scores)
    overall_avg = (math_avg + general_avg) / 2
    
    math_avg_base = mean([exact_match(p, g) for p, g in zip(base_preds['math'], golds['math'])])
    general_avg_base = mean([exact_match(p, g) for p, g in zip(base_preds['gen'], golds['gen'])])
    
    math_rel_imp = (math_avg - math_avg_base) / math_avg_base * 100
    gen_rel_drop = (general_avg_base - general_avg) / general_avg_base * 100
    return overall_avg, math_rel_imp, gen_rel_drop

Common pitfalls

• Reporting relative improvement/drop percentages (Math↑, Gen↓) instead of absolute scores, which obscures baseline performance differences across model sizes.
• Failing to remove math-related subsets from AGIEval and MMLU when computing GeneralAvg, which artificially inflates the catastrophic forgetting metric.
• Using the pre-training checkpoint instead of the instruct-tuned base model as the reference for relative metric calculations, violating the paper's defined baseline.

Evidence (verbatim from paper)

Math↑ denotes the relative MathAvg improvement over the instruct-tuned base within the same block: (M-M_base)/M_base×100. Gen↓ denotes the relative GeneralAvg drop: (G_base-G)/G_base×100. OverallAvg =(MathAvg+GeneralAvg)/2.

Citation

@misc{wang2026patch,
  title={Patch the Distribution Mismatch: RL Rewriting Agent for Stable Off-Policy SFT},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2602.11220}
}

• arXiv: 2602.11220