adu2021 Surgical Post-Training: Surgical Error Correction with Knowledge Preservation
Standard post-training data collection requires extensive high-quality annotation. Surgical Post-Training (SPoT) takes a different approach: starting with existing model-generated trajectories, use a stronger oracle model to surgically correct only the erroneous reasoning steps while preserving the original trajectory structure and style. This minimal-edit approach dramatically reduces annotation burden while maintaining knowledge from pretraining.
The core insight is that error correction should be surgical, not wholesale. Most of a trajectory is correct; fixing only the broken parts minimizes distribution shift. Combined with binary classification objectives and implicit KL regularization through reward-based loss, SPoT achieves consistent improvements with 4,000-10,000 rectified pairs instead of the 100,000+ examples typical approaches require.
Core Concept
SPoT operates through three coordinated mechanisms:
- Data Rectification Pipeline: Generate erroneous responses from the model, use an oracle to fix only the steps that fail, keep everything else unchanged
- Trajectory Similarity Enforcement: Filter pairs to ensure they differ only minimally (>60% token overlap), preventing major distribution shifts
- Binary Classification Objective: Treat correct vs. incorrect reasoning as a classification problem, then regularize via a KL constraint acting as an elastic tether to prevent catastrophic forgetting
Architecture Overview
- Input: Erroneous trajectories from base model, oracle model for correction
- Rectification: Oracle identifies step-level errors, proposes minimal fixes
- Similarity Filtering: Keep only pairs with high token-overlap ratio (LCS-based)
- Loss Design: Binary classification loss with KL-based regularization
- Output: Fine-tuned model with improved reasoning accuracy
Implementation Steps
Step 1: Generate candidate erroneous and correct pairs
Sample trajectories from the model, identify failures, and use oracle for correction.
def generate_rectified_pairs(base_model, oracle_model, prompt_batch,
num_samples_per_prompt=2):
"""Generate erroneous model outputs and oracle corrections."""
rectified_pairs = []
for prompt in prompt_batch:
# Sample multiple model outputs (some will likely be incorrect)
candidates = [base_model.generate(prompt) for _ in range(num_samples_per_prompt)]
for candidate in candidates:
# Check if candidate is correct via oracle evaluation
is_correct, error_steps = oracle_model.evaluate(candidate)
if not is_correct and len(error_steps) > 0:
# Oracle corrects only the identified error steps
corrected = oracle_model.fix_steps(candidate, error_steps)
rectified_pairs.append({
'incorrect': candidate,
'correct': corrected,
'error_steps': error_steps
})
return rectified_pairs
Step 2: Filter pairs by trajectory similarity
Ensure rectified pairs differ only minimally to avoid distribution shift. Use Longest Common Subsequence (LCS) to measure overlap.
def lcs_ratio(s1, s2):
"""Compute longest common subsequence length / max(len(s1), len(s2))."""
m, n = len(s1), len(s2)
dp = [[0] * (n + 1) for _ in range(m + 1)]
for i in range(1, m + 1):
for j in range(1, n + 1):
if s1[i - 1] == s2[j - 1]:
dp[i][j] = dp[i - 1][j - 1] + 1
else:
dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])
lcs_length = dp[m][n]
return lcs_length / max(m, n)
# Filter pairs: keep only those with >60% token-level overlap
min_similarity_ratio = 0.60
filtered_pairs = []
for pair in rectified_pairs:
tokens_incorrect = pair['incorrect'].split()
tokens_correct = pair['correct'].split()
similarity = lcs_ratio(tokens_incorrect, tokens_correct)
if similarity >= min_similarity_ratio:
filtered_pairs.append(pair)
# else: too much divergence; skip to avoid distribution shift
print(f"Kept {len(filtered_pairs)} pairs after similarity filtering")
Step 3: Tokenize and prepare for training
Convert trajectories to token sequences for model training.
def prepare_training_batch(filtered_pairs, tokenizer, max_length=512):
"""Tokenize rectified pairs for training."""
incorrect_ids = []
correct_ids = []
attention_masks = []
for pair in filtered_pairs:
# Tokenize both incorrect and correct versions
incorrect_tokens = tokenizer.encode(
pair['incorrect'],
max_length=max_length,
truncation=True,
padding='max_length',
return_tensors='pt'
)
correct_tokens = tokenizer.encode(
pair['correct'],
max_length=max_length,
truncation=True,
padding='max_length',
return_tensors='pt'
)
incorrect_ids.append(incorrect_tokens['input_ids'])
correct_ids.append(correct_tokens['input_ids'])
attention_masks.append(correct_tokens['attention_mask'])
return {
'incorrect_ids': torch.cat(incorrect_ids),
'correct_ids': torch.cat(correct_ids),
'attention_mask': torch.cat(attention_masks)
}
Step 4: Design binary classification loss with KL regularization
The core loss treats correct vs. incorrect as classification, regularized by KL to reference model.
def compute_spt_loss(model, reference_model, batch, temperature=1.0):
"""
SPoT loss = Binary Classification + KL Regularization
Classification: maximize likelihood of correct responses, minimize incorrect
KL: prevent divergence from reference model via elastic tether
"""
incorrect_ids = batch['incorrect_ids']
correct_ids = batch['correct_ids']
attention_mask = batch['attention_mask']
# Forward pass: get log probabilities for both sequences
with torch.no_grad():
ref_logits_correct = reference_model(correct_ids)['logits']
ref_logits_incorrect = reference_model(incorrect_ids)['logits']
logits_correct = model(correct_ids, attention_mask=attention_mask)['logits']
logits_incorrect = model(incorrect_ids, attention_mask=attention_mask)['logits']
# Binary classification objective:
# - Maximize likelihood of correct trajectory
# - Minimize likelihood of incorrect trajectory
loss_correct = -torch.nn.functional.log_softmax(logits_correct, dim=-1).mean()
loss_incorrect = torch.nn.functional.log_softmax(logits_incorrect, dim=-1).mean()
classification_loss = loss_correct - loss_incorrect # Rank correct above incorrect
# KL regularization: constrain divergence from reference model
# This acts as an "elastic tether" preventing catastrophic forgetting
kl_loss = torch.nn.functional.kl_div(
torch.nn.functional.log_softmax(logits_correct / temperature, dim=-1),
torch.nn.functional.softmax(ref_logits_correct / temperature, dim=-1),
reduction='batchmean'
)
# Combined loss with KL weight
beta = 0.1 # KL regularization strength
total_loss = classification_loss + beta * kl_loss
return total_loss
Step 5: Training loop with convergence monitoring
Train the model with rectified pairs, monitoring both accuracy improvement and knowledge retention.
def train_spt(model, reference_model, train_dataloader,
epochs=3, learning_rate=2e-5):
"""Fine-tune model with Surgical Post-Training."""
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
for epoch in range(epochs):
total_loss = 0.0
num_batches = 0
for batch in train_dataloader:
optimizer.zero_grad()
# Compute loss
loss = compute_spt_loss(model, reference_model, batch)
# Backward and optimize
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
total_loss += loss.item()
num_batches += 1
avg_loss = total_loss / num_batches
print(f"Epoch {epoch + 1}: Loss = {avg_loss:.4f}")
# Early stopping: if loss plateaus, reduce learning rate
if epoch > 0 and abs(avg_loss - prev_loss) < 1e-4:
learning_rate *= 0.5
for param_group in optimizer.param_groups:
param_group['lr'] = learning_rate
prev_loss = avg_loss
return model
Step 6: Evaluation and knowledge retention check
Verify that fine-tuned model improves on error correction while retaining general knowledge.
def evaluate_spt(model, test_prompts, oracle_model, baseline_model):
"""Evaluate SPoT: accuracy improvement and knowledge retention."""
correct_before = 0
correct_after = 0
total = len(test_prompts)
for prompt in test_prompts:
# Baseline model output
response_baseline = baseline_model.generate(prompt)
is_correct_before, _ = oracle_model.evaluate(response_baseline)
# Fine-tuned model output
response_ft = model.generate(prompt)
is_correct_after, _ = oracle_model.evaluate(response_ft)
if is_correct_before:
correct_before += 1
if is_correct_after:
correct_after += 1
improvement = (correct_after - correct_before) / total * 100
accuracy_after = correct_after / total * 100
print(f"Accuracy improvement: {improvement:+.1f}%")
print(f"Final accuracy: {accuracy_after:.1f}%")
return improvement, accuracy_after
Practical Guidance
Hyperparameter Selection:
- Similarity ratio threshold: 0.6 (60% LCS overlap). Lower = more change tolerance; higher = stricter conservation.
- KL weight β: 0.05-0.2. Higher = stronger forgetting prevention; too high can prevent improvement.
- Temperature for KL: 1.0 standard; increase to 2-3 for softer regularization on divergent tasks.
- Training epochs: 2-5. More epochs risk overfitting to rectified data; SPoT gains saturate quickly.
- Learning rate: 2e-5 (conservative). Lower rates preserve pretraining better.
When to Use:
- Post-training with limited annotation budget (has oracle but few labeled examples)
- Correcting specific reasoning errors in code generation, math, planning
- Fine-tuning after pretraining when preserving general knowledge is critical
When NOT to Use:
- Complete retraining (use standard SFT instead)
- Tasks where oracle corrections require major trajectory changes
- Settings with abundant high-quality labeled data (standard approaches are more direct)
Common Pitfalls:
- Broken rectification pipeline: Oracle corrections must be faithful and minimal. Poor oracle → noise in training.
- Over-filtering by similarity: Too strict LCS threshold discards useful corrections. Start at 0.6, lower gradually.
- KL regularization too strong: β > 0.3 can prevent improvement. Monitor validation accuracy; increase β if it drops.
- Insufficient rectified data: <1000 pairs generally insufficient for stable training. Collect 4,000-10,000 for robust gains.