Profile My skills Saved Collections Edit profile Submit a skill

Swe Rm Verifier

Improve SWE agent RL via execution-free reward models optimized for three metrics beyond TTS: discriminative ability (AUC) and calibration (ECE). Shows TTS alone insufficient—models must distinguish correct/incorrect trajectories and align confidence with actual correctness—improving SWE-Bench Verified by 7-10 points with 30B MoE verifier.

adu2021 fa22b9e 1022 B Updated
AI & ML

File contents

Overview

SWE-RM shows that reward model evaluation requires three complementary metrics.

Core Technique

Three-Metric Evaluation:

# Not just TTS (top-1 ranking)
tts_score = rank_best_solution_first(predictions)

# Also discriminative ability
auc_score = compute_auc(correct_vs_incorrect)

# And calibration
ece_score = expected_calibration_error(confidence, accuracy)

When to Use

Use when: SWE agent training, RL reward modeling, importance of calibration.

References

  • Discriminative ability vs ranking metrics
  • Calibration error measurement
  • Multi-objective reward model design

adu2021/skillxiv/tree/main/skills/skillxiv-v0.0.2-claude-opus-4.6/swe-rm-verifier commit fa22b9e0ce

Browse category

Frequently asked questions

npx skillmds@latest add adu2021/swe-rm-verifier