Context Compression Evaluation Framework

Context Compression Evaluation Framework

This document provides the complete evaluation framework for measuring context compression quality, including probe types, scoring rubrics, and LLM judge configuration.

Probe Types

Recall Probes

Test factual retention of specific details from conversation history.

Structure:

Question: [Ask for specific fact from truncated history]
Expected: [Exact detail that should be preserved]
Scoring: Match accuracy of technical details

Examples:

• "What was the original error message that started this debugging session?"
• "What version of the dependency did we decide to use?"
• "What was the exact command that failed?"

Artifact Probes

Test file tracking and modification awareness.

Structure:

Question: [Ask about files created, modified, or examined]
Expected: [Complete list with change descriptions]
Scoring: Completeness of file list and accuracy of change descriptions

Examples:

• "Which files have we modified? Describe what changed in each."
• "What new files did we create in this session?"
• "Which configuration files did we examine but not change?"

Continuation Probes

Test ability to continue work without re-fetching context.

Structure:

Question: [Ask about next steps or current state]
Expected: [Actionable next steps based on session history]
Scoring: Ability to continue without requesting re-read of files

Examples:

• "What should we do next?"
• "What tests are still failing and why?"
• "What was left incomplete from our last step?"

Decision Probes

Test retention of reasoning chains and decision rationale.

Structure:

Question: [Ask about why a decision was made]
Expected: [Reasoning that led to the decision]
Scoring: Preservation of decision context and alternatives considered

Examples:

• "We discussed options for the Redis issue. What did we decide and why?"
• "Why did we choose connection pooling over per-request connections?"
• "What alternatives did we consider for the authentication fix?"

Scoring Rubrics

Accuracy Dimension

Context Awareness Dimension

Artifact Trail Dimension

Completeness Dimension

Continuity Dimension

Instruction Following Dimension

LLM Judge Configuration

System Prompt

You are an expert evaluator assessing AI assistant responses in software development conversations.

Your task is to grade responses against specific rubric criteria. For each criterion:
1. Read the criterion question carefully
2. Examine the response for evidence
3. Assign a score from 0-5 based on the scoring guide
4. Provide brief reasoning for your score

Be objective and consistent. Focus on what is present in the response, not what could have been included.

Judge Input Format

{
  "probe_question": "What was the original error message?",
  "model_response": "[Response to evaluate]",
  "compacted_context": "[The compressed context that was provided]",
  "ground_truth": "[Optional: known correct answer]",
  "rubric_criteria": ["accuracy_factual", "accuracy_technical", "context_conversation_state"]
}

Judge Output Format

{
  "criterionResults": [
    {
      "criterionId": "accuracy_factual",
      "score": 5,
      "reasoning": "Response correctly identifies the 401 error, specific endpoint, and root cause."
    }
  ],
  "aggregateScore": 4.8,
  "dimensionScores": {
    "accuracy": 4.9,
    "context_awareness": 4.5,
    "artifact_trail": 3.2,
    "completeness": 5.0,
    "continuity": 4.8,
    "instruction_following": 5.0
  }
}

Benchmark Results Reference

Performance across compression methods (based on 36,000+ messages):

Key Findings:

1. Accuracy gap: 0.61 points between best and worst methods
2. Context awareness gap: 0.45 points, favoring anchored iterative
3. Artifact trail: Universally weak (2.19-2.45), needs specialized handling
4. Completeness and instruction following: Minimal differentiation

Statistical Considerations

• Differences of 0.26-0.35 points are consistent across task types and session lengths
• Pattern holds for both short and long sessions
• Pattern holds across debugging, feature implementation, and code review tasks
• Sample size: 36,611 messages across hundreds of compression points

Implementation Notes

Probe Generation

Generate probes at each compression point based on truncated history:

1. Extract factual claims for recall probes
2. Extract file operations for artifact probes
3. Extract incomplete tasks for continuation probes
4. Extract decision points for decision probes

Grading Process

1. Feed probe question + model response + compressed context to judge
2. Evaluate against each criterion in rubric
3. Output structured JSON with scores and reasoning
4. Compute dimension scores as weighted averages
5. Compute overall score as unweighted average of dimensions

Blinding

The judge should not know which compression method produced the response being evaluated. This prevents bias toward known methods.