adu2021 Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback
Core Concept
Despite access to high-quality feedback from more capable models with ground-truth knowledge, large language models consistently show resistance to incorporating guidance to improve responses. This "Feedback Friction" dominates 62.8-100% of unsolved problems even under ideal conditions (frontier models, extended thinking, multiple feedback iterations). The study reveals that model confidence (measured via semantic entropy) predicts feedback receptiveness, and that feedback resistance—not feedback quality—is the limiting factor.
Architecture Overview
- Systematic Feedback Framework: Three mechanisms of increasing sophistication (binary correctness, self-generated reflection, strong-model reflection) across up to 10 iterations
- Monotonic Accuracy Measurement: Models retain correct answers while only modifying incorrect responses, ensuring accurate tracking
- Semantic Entropy Analysis: Measures true confidence at meaning level rather than surface tokens; high correlation with feedback receptiveness
- Error Categorization: Three failure types identified; feedback resistance dominates across diverse benchmarks
- Tested Models: Llama-3.3-70B, Llama-4, Claude 3.7, Claude 3.7 Thinking on AIME, MATH-500, GPQA, MMLU variants
Implementation
Step 1: Feedback Framework and Mechanisms
import torch
from typing import Dict, List, Tuple
class FeedbackMechanism:
"""
Three levels of feedback sophistication for guiding LLM improvements.
Enables systematic study of feedback receptiveness.
"""
def __init__(self, model, ground_truth_answers):
self.model = model
self.ground_truth = ground_truth_answers
self.feedback_history = []
def feedback_mechanism_1(self, question: str, model_response: str) -> str:
"""
F₁: Simple binary correctness signal.
Returns: "Your answer is correct" or "Your answer is incorrect"
"""
expected_answer = self.ground_truth.get(question)
if self._extract_answer(model_response) == expected_answer:
return "Your answer is correct. Stop here."
else:
return "Your answer is incorrect. Please reconsider."
def feedback_mechanism_2(self, question: str, model_response: str) -> str:
"""
F₂: Self-generated reflective feedback.
Model generates its own analysis of why the response might be wrong.
"""
reflection_prompt = f"""
Question: {question}
Your response: {model_response}
Analyze your reasoning step-by-step:
1. What assumption did you make?
2. Could there be an error in your calculation?
3. Did you consider all constraints?
Provide honest assessment of correctness.
"""
# Model generates self-reflection
self_reflection = self.model.generate(reflection_prompt, max_length=300)
return f"Self-analysis: {self_reflection}. Please revise your answer."
def feedback_mechanism_3(self, question: str, model_response: str) -> str:
"""
F₃: Strong-model reflective feedback.
GPT-4.1 mini provides expert analysis with access to ground-truth.
"""
expected_answer = self.ground_truth.get(question)
feedback_prompt = f"""
Question: {question}
Model's response: {model_response}
Expected answer: {expected_answer}
Provide guidance on what went wrong and how to improve:
1. Identify the error
2. Suggest correct approach
3. Point out missed considerations
"""
# Strong model provides feedback
expert_feedback = self.model.generate(feedback_prompt, max_length=300)
return expert_feedback
def apply_iterative_feedback(self, question: str, max_iterations: int = 10) -> Dict:
"""
Apply feedback iteratively up to max_iterations.
Track whether model incorporates guidance over time.
"""
current_response = self.model.generate(question, max_length=512)
iteration_results = []
for iteration in range(max_iterations):
# Evaluate current response
expected_answer = self.ground_truth.get(question)
is_correct = self._extract_answer(current_response) == expected_answer
iteration_results.append({
'iteration': iteration,
'is_correct': is_correct,
'response': current_response,
'confidence': self._measure_confidence(current_response)
})
if is_correct:
# Correct answer found; stop iteration
break
# Select feedback mechanism
feedback_level = min(iteration // 3, 2) # Escalate feedback
if feedback_level == 0:
feedback = self.feedback_mechanism_1(question, current_response)
elif feedback_level == 1:
feedback = self.feedback_mechanism_2(question, current_response)
else:
feedback = self.feedback_mechanism_3(question, current_response)
# Apply feedback and generate new response
guided_prompt = f"""
Question: {question}
Previous response: {current_response}
Feedback: {feedback}
Based on this feedback, please provide a new, improved response:
"""
current_response = self.model.generate(guided_prompt, max_length=512)
self.feedback_history.append({
'iteration': iteration,
'feedback_level': feedback_level,
'feedback': feedback
})
return {
'question': question,
'iterations': iteration_results,
'final_correct': iteration_results[-1]['is_correct'] if iteration_results else False
}
def _extract_answer(self, response: str) -> str:
"""Extract final answer from response."""
# Simplified: would use answer extraction based on task type
lines = response.strip().split('\n')
return lines[-1] if lines else ""
def _measure_confidence(self, response: str) -> float:
"""Measure model confidence in response (0-1)."""
# Placeholder: actual implementation uses semantic entropy
return 0.5
Step 2: Semantic Entropy for Confidence Measurement
import numpy as np
from collections import Counter
class SemanticEntropyAnalyzer:
"""
Measures true model confidence using semantic entropy.
Operates at meaning level (multiple samples) rather than surface tokens.
Predicts feedback receptiveness.
"""
def __init__(self, model, num_samples=50):
self.model = model
self.num_samples = num_samples
def compute_semantic_entropy(self, question: str) -> float:
"""
Compute semantic entropy: uncertainty about meaning, not tokens.
Procedure:
1. Generate multiple samples from model
2. Group semantically equivalent responses
3. Compute entropy over meaning clusters
"""
# Generate multiple samples
samples = []
for _ in range(self.num_samples):
sample = self.model.generate(question, max_length=512, temperature=0.7)
samples.append(sample)
# Cluster semantically equivalent responses
clusters = self._cluster_semantic_groups(samples)
# Compute entropy over clusters
cluster_sizes = [len(c) for c in clusters]
probabilities = np.array(cluster_sizes) / len(samples)
# Shannon entropy
entropy = -np.sum(probabilities * np.log(probabilities + 1e-10))
return entropy
def _cluster_semantic_groups(self, samples: List[str]) -> List[List[str]]:
"""
Cluster samples into semantically equivalent groups.
Uses answer extraction + embedding similarity.
"""
# Extract answers
answers = [self._extract_answer(s) for s in samples]
# Simple clustering: group identical answers
answer_groups = {}
for answer, sample in zip(answers, samples):
if answer not in answer_groups:
answer_groups[answer] = []
answer_groups[answer].append(sample)
return list(answer_groups.values())
def predict_feedback_receptiveness(self, semantic_entropy: float) -> float:
"""
Predict probability model will accept feedback based on entropy.
Key finding: Absolute improvement rate increases from ~0 at low entropy
(high confidence, resistant to feedback) to 0.4-0.8 at high entropy
(low confidence, receptive to feedback).
"""
# Empirically derived relationship
if semantic_entropy < 0.5:
receptiveness = 0.05 # High confidence: very resistant
elif semantic_entropy < 1.0:
receptiveness = 0.15
elif semantic_entropy < 1.5:
receptiveness = 0.4 # Medium confidence
else:
receptiveness = 0.7 # Low confidence: receptive
return receptiveness
def _extract_answer(self, response: str) -> str:
"""Extract answer for semantic clustering."""
return response.strip().split('\n')[-1] if response else ""
Step 3: Error Analysis and Categorization
class FeedbackResistanceAnalyzer:
"""
Categorizes persistent failures into three error types.
Identifies whether failures are due to feedback quality or feedback resistance.
"""
def __init__(self, model):
self.model = model
def categorize_persistent_failure(self, question: str, feedback_history: List,
max_iterations: int = 10) -> Dict:
"""
Analyze why a problem remains unsolved after feedback iterations.
Three error categories:
1. Feedback Quality Issue: Feedback is incorrect or unhelpful
2. Comprehension Issue: Model understands feedback but can't apply it
3. Feedback Resistance: Model ignores feedback despite understanding
"""
categories = {
'feedback_quality': 0.0,
'comprehension': 0.0,
'resistance': 0.0
}
# Test 1: Verify feedback correctness
feedback_quality_score = self._verify_feedback_correctness(question, feedback_history)
categories['feedback_quality'] = feedback_quality_score
# Test 2: Model comprehension of feedback
comprehension_score = self._test_comprehension(question, feedback_history[-1])
categories['comprehension'] = comprehension_score
# Test 3: Feedback resistance (residual)
resistance_score = 1.0 - feedback_quality_score - comprehension_score
categories['resistance'] = max(0, resistance_score)
# Normalize
total = sum(categories.values())
if total > 0:
for key in categories:
categories[key] /= total
return categories
def _verify_feedback_correctness(self, question: str, feedback: List) -> float:
"""
Verify that provided feedback is actually correct.
Returns confidence that feedback accurately guides to correct answer.
"""
verification_prompt = f"""
Question: {question}
Feedback provided:
{feedback[-1] if feedback else "None"}
Is this feedback correct and helpful? (0-1)
"""
# Simplified: assume feedback from GPT-4 is correct
return 0.95
def _test_comprehension(self, question: str, feedback: str) -> float:
"""
Test whether model actually understands the feedback.
Ask model to explain what it should do based on feedback.
"""
comprehension_prompt = f"""
Question: {question}
Feedback: {feedback}
Based on this feedback, what is the correct approach? Explain:
"""
explanation = self.model.generate(comprehension_prompt, max_length=300)
# Score explanation against feedback content
comprehension_score = self._score_explanation(explanation, feedback)
return comprehension_score
def _score_explanation(self, explanation: str, feedback: str) -> float:
"""Score how well explanation reflects feedback understanding."""
# Placeholder: would use semantic similarity
return 0.5
Step 4: Mitigation Strategies
class FeedbackFrictionMitigation:
"""
Explores strategies to reduce feedback resistance.
Tests: temperature scaling, rejection sampling, combined approaches.
"""
def __init__(self, model):
self.model = model
def mitigate_with_temperature_scaling(self, question: str, feedback: str,
temperatures: List[float]) -> Tuple[bool, float]:
"""
Strategy 1: Progressive temperature increase encourages reconsideration.
Higher temperature = more exploration, potentially better feedback incorporation.
"""
best_response = None
best_accuracy = 0.0
for temp in temperatures:
guided_prompt = f"""
Question: {question}
Feedback: {feedback}
Please revise your answer:
"""
response = self.model.generate(
guided_prompt,
temperature=temp,
max_length=512
)
# Evaluate response
is_correct = self._evaluate(response, question)
if is_correct:
return True, temp
if self._response_quality(response) > best_accuracy:
best_response = response
best_accuracy = self._response_quality(response)
return False, None # Mitigation unsuccessful
def mitigate_with_rejection_sampling(self, question: str, feedback: str,
num_samples: int = 10) -> Tuple[bool, float]:
"""
Strategy 2: Rejection sampling - generate multiple responses, select best.
Combines temperature variation with explicit selection.
"""
candidates = []
for temp in [0.3, 0.7, 1.0, 1.3]:
for _ in range(num_samples // 4):
guided_prompt = f"""
Question: {question}
Feedback: {feedback}
Please provide a revised answer:
"""
response = self.model.generate(
guided_prompt,
temperature=temp,
max_length=512
)
candidates.append({
'response': response,
'correct': self._evaluate(response, question),
'quality': self._response_quality(response)
})
# Select best candidate
best_candidate = max(candidates, key=lambda x: x['quality'])
if best_candidate['correct']:
return True, best_candidate['quality']
return False, best_candidate['quality']
def mitigate_combined_approach(self, question: str, feedback: str,
max_attempts: int = 5) -> Tuple[bool, float]:
"""
Strategy 3: Combined temperature scaling + rejection sampling.
Escalates intervention based on failure analysis.
"""
for attempt in range(max_attempts):
# Attempt 1-2: Temperature scaling
if attempt < 2:
success, score = self.mitigate_with_temperature_scaling(
question, feedback, [0.5 + 0.3 * attempt]
)
if success:
return True, score
# Attempt 3-4: Rejection sampling
else:
success, score = self.mitigate_with_rejection_sampling(
question, feedback, num_samples=20
)
if success:
return True, score
return False, 0.0
def _evaluate(self, response: str, question: str) -> bool:
"""Check if response is correct."""
# Placeholder implementation
return False
def _response_quality(self, response: str) -> float:
"""Score response quality 0-1."""
# Placeholder implementation
return 0.5
Practical Guidance
Feedback Framework Implementation:
- F₁ (binary): Quick, no model required; use first
- F₂ (self-reflection): Model-generated; captures own reasoning limitations
- F₃ (strong-model): GPT-4 mini; most informative but costly
- Escalate feedback mechanism over iterations (start F₁, progress to F₃)
Confidence Measurement:
- Semantic entropy threshold: entropy < 0.5 indicates high confidence (resistant)
- Entropy > 1.5 indicates low confidence (receptive to feedback)
- Sample 50 outputs per question for robust entropy estimate
Error Attribution:
- Feedback resistance dominates: 62.8-100% of unsolved problems
- Feedback quality: Usually good (95%+ from GPT-4)
- Comprehension: Mixed; models understand but resist applying
Mitigation Effectiveness:
- Temperature scaling: Modest improvements; max +5% accuracy
- Rejection sampling: Better; +10-15% improvement
- Combined approach: Best; +15-20% improvement, but still falls short of theoretical ceiling
When Feedback Friction Occurs:
- Reasoning tasks (AIME, MATH, GPQA): High friction
- Factual tasks (MMLU): Lower friction
- Confident predictions: Higher friction
- Extended thinking models: Still shows significant friction
Recommendations:
- Accept that LLMs won't achieve theoretical maximum via iterative feedback
- Budget resources for rejection sampling (most effective practical mitigation)
- Use semantic entropy to identify which problems are worth improving
- Focus feedback on low-confidence responses where resistance is lower
Reference
- Semantic entropy: Meaning-level uncertainty vs token-level entropy
- Feedback friction: Resistance to incorporating guidance despite understanding
- Rejection sampling: Select best from multiple samples; often more effective than resampling with guidance
- Iterative refinement limits: Fundamental limitation in current LLM architecture for feedback incorporation