Inplace Feedback Multi Turn Reasoning

In-Place Feedback: Direct Response Editing for Precise Guidance

Traditional multi-turn feedback works by issuing separate corrections: "Your answer has an error on step 3. Here's the correct step." The model then tries to apply this external feedback to its previous response. The problem is that external feedback is ambiguous—the model often fails to apply corrections precisely, leaving errors uncorrected or introducing new mistakes.

In-Place Feedback inverts this: users directly edit the model's response text, marking corrections inline. The model then conditions on this edited version, receiving explicit guidance on where and how to fix errors. This dramatically improves correction accuracy while reducing token overhead.

Core Concept

In-Place Feedback operates in two modes:

1. Annotation mode: User (or automatic corrector) edits model's response in-place, marking corrections
2. Generation mode: Model receives the edited version as context and generates an improved response

The edited response acts as a concrete example of what the model should have done, guiding generation more effectively than abstract feedback text.

Architecture Overview

• Response generator: Initial response from the model
• Editor: Marks errors and corrections in the response (human or automated)
• Context encoder: Converts edited response to input for next generation
• Guided generator: Conditions on edited context to produce corrected response
• Feedback tracker: Monitors which types of errors are fixed vs. persisting

Implementation Steps

First, implement response annotation (marking errors):

from dataclasses import dataclass
from typing import List, Tuple

@dataclass
class Annotation:
    """Mark errors and corrections in a response."""
    start_idx: int           # Character position of error start
    end_idx: int             # Character position of error end
    original_text: str       # Original (incorrect) text
    corrected_text: str      # Corrected text
    error_type: str          # "arithmetic", "logic", "retrieval", etc.
    confidence: float        # 0-1 confidence in correction

class ResponseEditor:
    """
    Edit model responses by marking corrections inline.
    """
    def __init__(self, error_detector):
        self.error_detector = error_detector

    def annotate_response(self, response: str, problem: str, ground_truth: str):
        """
        Identify and mark errors in response.

        Args:
            response: Model's response text
            problem: Original problem statement
            ground_truth: Correct answer or reference

        Returns:
            annotations: List of errors to correct
            edited_response: Response with corrections marked
        """
        annotations = []

        # Detect errors using error detector (could be learned model, rule-based, etc.)
        errors = self.error_detector.find_errors(response, problem, ground_truth)

        # Create edited response with corrections
        edited_response = response
        offset = 0  # Track position offset as we edit

        for error in errors:
            original_text = error["original_text"]
            corrected_text = error["corrected_text"]
            error_pos = error["position"]

            # Adjust position for previous edits
            adjusted_pos = error_pos + offset

            # Create annotation
            annotation = Annotation(
                start_idx=adjusted_pos,
                end_idx=adjusted_pos + len(original_text),
                original_text=original_text,
                corrected_text=corrected_text,
                error_type=error.get("type", "unknown"),
                confidence=error.get("confidence", 0.9)
            )
            annotations.append(annotation)

            # Apply correction to edited response
            edited_response = (
                edited_response[:adjusted_pos] +
                corrected_text +
                edited_response[adjusted_pos + len(original_text):]
            )

            # Update offset
            offset += len(corrected_text) - len(original_text)

        return annotations, edited_response

    def format_edited_response_for_input(self, edited_response: str, annotations: List[Annotation]):
        """
        Format the edited response as input for the next generation step.

        Args:
            edited_response: Fully corrected response
            annotations: List of corrections applied

        Returns:
            formatted_input: Text showing what was corrected
        """
        # Format: show original | corrected pairs for clarity
        formatted = "Previous response with corrections:\n\n"

        current_pos = 0
        for annotation in annotations:
            # Text before this annotation (unchanged)
            if annotation.start_idx > current_pos:
                formatted += edited_response[current_pos:annotation.start_idx]

            # Mark the correction
            formatted += (
                f"[CORRECTED: '{annotation.original_text}' -> '{annotation.corrected_text}']"
            )

            current_pos = annotation.end_idx

        # Text after last annotation
        if current_pos < len(edited_response):
            formatted += edited_response[current_pos:]

        return formatted

Now implement guided generation conditioned on edited response:

class GuidedGenerator:
    """
    Generate improved response conditioned on corrections.
    """
    def __init__(self, model):
        self.model = model

    def generate_corrected_response(self, problem: str, original_response: str,
                                    edited_response: str, annotations: List[Annotation]):
        """
        Generate a corrected response using edited version as context.

        Args:
            problem: Original problem statement
            original_response: Model's initial response
            edited_response: Response with corrections marked
            annotations: List of corrections applied

        Returns:
            corrected_response: Improved response
        """
        # Build prompt showing what needs fixing
        formatted_edits = self._format_edits(annotations)

        prompt = f"""You previously attempted this problem:

Problem: {problem}

Your previous response:
{original_response}

Corrections needed:
{formatted_edits}

Generate an improved response that incorporates these corrections:"""

        # Generate corrected response
        corrected = self.model.generate(prompt, max_length=500)

        return corrected

    def _format_edits(self, annotations: List[Annotation]) -> str:
        """Format annotations for display to the model."""
        if not annotations:
            return "[No corrections needed]"

        formatted = []
        for i, ann in enumerate(annotations, 1):
            formatted.append(
                f"{i}. {ann.error_type}: '{ann.original_text}' should be '{ann.corrected_text}'"
            )

        return "\n".join(formatted)

    def iterative_correction(self, problem: str, response: str, error_detector,
                            num_iterations=3, min_confidence=0.7):
        """
        Iteratively correct response through multiple passes.

        Args:
            problem: Problem statement
            response: Initial response
            error_detector: Error detection model
            num_iterations: How many correction rounds
            min_confidence: Only correct errors above this confidence

        Returns:
            final_response: Final corrected response
        """
        current_response = response

        for iteration in range(num_iterations):
            # Detect errors in current response
            editor = ResponseEditor(error_detector)
            annotations, edited_response = editor.annotate_response(
                current_response,
                problem,
                None  # No ground truth for self-correction
            )

            # Filter out low-confidence corrections
            high_confidence = [a for a in annotations if a.confidence >= min_confidence]

            if not high_confidence:
                break  # No clear errors to fix

            # Generate corrected version
            current_response = self.generate_corrected_response(
                problem,
                current_response,
                edited_response,
                high_confidence
            )

        return current_response

Implement the full interactive correction loop:

class InPlaceFeedbackSystem:
    """
    Complete system for in-place feedback on multi-turn reasoning.
    """
    def __init__(self, model, error_detector):
        self.model = model
        self.editor = ResponseEditor(error_detector)
        self.generator = GuidedGenerator(model)
        self.feedback_history = []

    def single_turn_with_feedback(self, problem: str, max_turns: int = 3):
        """
        Single problem with interactive correction feedback.

        Args:
            problem: Problem to solve
            max_turns: Maximum feedback iterations

        Returns:
            final_response: Final corrected response
            history: Interaction history
        """
        # Initial response
        response = self.model.generate(problem)
        self.feedback_history.append({
            "turn": 0,
            "response": response,
            "annotations": [],
            "tokens_used": len(response.split())
        })

        for turn in range(1, max_turns + 1):
            # Detect errors in current response
            # (In practice, human or oracle would provide feedback)
            annotations, edited_response = self.editor.annotate_response(
                response,
                problem,
                None  # Auto-detection; could be human-provided
            )

            if not annotations:
                break  # No errors detected

            # Generate improved response with in-place feedback
            response = self.generator.generate_corrected_response(
                problem,
                response,
                edited_response,
                annotations
            )

            self.feedback_history.append({
                "turn": turn,
                "response": response,
                "annotations": annotations,
                "tokens_used": len(response.split())
            })

        return response, self.feedback_history

    def compare_token_efficiency(self):
        """
        Compare in-place feedback vs. traditional separate feedback.

        Returns:
            comparison: Token usage and accuracy comparison
        """
        total_tokens_inplace = sum(h["tokens_used"] for h in self.feedback_history)

        # Traditional approach: separate feedback takes more tokens
        # (feedback text + full regeneration, not incremental)
        estimated_separate_tokens = total_tokens_inplace * 1.79  # ~79% overhead

        return {
            "inplace_tokens": total_tokens_inplace,
            "separate_tokens": estimated_separate_tokens,
            "savings_percent": (1 - total_tokens_inplace / estimated_separate_tokens) * 100
        }

Implement automatic error detection for self-correction:

class AutomaticErrorDetector:
    """
    Detect errors automatically for self-correction scenarios.
    """
    def __init__(self, verifier_model):
        self.verifier = verifier_model

    def find_errors(self, response: str, problem: str, ground_truth: str = None):
        """
        Identify errors in response automatically.

        Args:
            response: Model response to analyze
            problem: Problem statement
            ground_truth: Optional correct answer for validation

        Returns:
            errors: List of detected errors with corrections
        """
        errors = []

        # Parse response into steps/statements
        statements = self._parse_response(response)

        for stmt_idx, statement in enumerate(statements):
            # Check each statement for potential errors
            is_valid = self.verifier.check_statement(statement, problem)

            if not is_valid:
                # Generate correction
                correction = self.verifier.suggest_correction(statement, problem)

                error = {
                    "position": self._find_position_in_response(response, statement),
                    "original_text": statement,
                    "corrected_text": correction,
                    "type": self.verifier.get_error_type(statement),
                    "confidence": self.verifier.get_confidence(statement)
                }
                errors.append(error)

        return errors

    def _parse_response(self, response: str) -> List[str]:
        """Split response into analyzable statements."""
        # Simple heuristic: split on periods/newlines
        statements = [s.strip() for s in response.split('.') if s.strip()]
        return statements

    def _find_position_in_response(self, response: str, statement: str) -> int:
        """Find character position of statement in response."""
        # Handle partial matches
        idx = response.find(statement)
        return idx if idx >= 0 else 0

Practical Guidance

When to use In-Place Feedback:

• Interactive reasoning tasks where users provide corrections
• Self-correction scenarios with automatic error detection
• Multi-turn QA where precision of fixes matters
• Low-latency settings (saves tokens = faster response)

When NOT to use:

• Single-turn generation (no feedback loop)
• Domains without clear error detection
• Fully automatic workflows (human editing not available)
• Tasks where 79% token savings is negligible (compute-rich setting)

Token efficiency comparison:

Accuracy improvements on reasoning tasks:

Key hyperparameters:

Common pitfalls:

• Cascade errors: If first correction is wrong, second round amplifies it. Validate error detection accuracy >85%.
• Over-correction: Model becomes "correction-seeking" (generates intentionally flawed responses to trigger feedback). Use entropy regularization to prevent.
• Feedback clarity: Formatted edits must be unambiguous. Test on 20 examples to ensure model correctly parses corrections.
• Convergence failure: Sometimes iterations increase error rate. Cap iterations and use best-seen response if later responses worse.

Integration checklist:

• Implement error detector; validate on 100 examples (accuracy >85%)
• Test formatted feedback on 20 examples; verify model applies corrections
• Run single-turn vs. in-place feedback comparison on test set
• Measure token overhead: should be ~50% less than traditional feedback
• Monitor accuracy improvement per turn (should be monotonic or plateau)
• Evaluate on multi-turn reasoning tasks (math, logic, code)

Reference: https://arxiv.org/abs/2510.00777