Semi Dynamic Context Compression Eval

semi-dynamic-context-compression-eval

Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio — Yijiong Yu et al. (arXiv:2603.25926, 2026)

What this evaluates

Evaluates the ability of context compression methods to preserve information density for downstream reading comprehension tasks. It probes whether adaptive, density-aware compression can maintain answer accuracy while significantly reducing context length compared to static baselines.

Datasets

• HotpotQA — total 1000; splits: test (1000)
• SQuAD — total 1000; splits: test (1000)
• Natural Questions — total 1000; splits: test (1000)
• AdversarialQA — total 1000; splits: test (1000)

Metrics

• substring accuracy (primary) — range: [0, 1]
  • A score of 1 is awarded if the exact reference answer appears anywhere as a substring within the generated output, else 0. Averaged over all test instances.
• average compression ratio — range: other
  • Sum of the original context lengths of all correctly answered samples divided by the sum of their compressed lengths. Only instances answered correctly are included in the calculation.

Input / output format

Input: Context passages (filtered to < 2,048 tokens) and corresponding questions/tasks. The model receives the context after it has been compressed via a mean-pooling encoder.

Output: Generated text/answer string.

Scoring recipe

predictions = model.generate(compressed_context, question)
golds = [sample['answer'] for sample in dataset]
orig_lens = [sample['orig_len'] for sample in dataset]
comp_lens = [sample['comp_len'] for sample in dataset]

correct_mask = [gold in pred for pred, gold in zip(predictions, golds)]
accuracy = sum(correct_mask) / len(correct_mask)

correct_orig = [o for o, c in zip(orig_lens, comp_lens) if correct]
correct_comp = [c for o, c in zip(orig_lens, comp_lens) if correct]
if correct_comp:
    avg_comp_ratio = sum(correct_orig) / sum(correct_comp)
else:
    avg_comp_ratio = 0

Common pitfalls

• Aggressively compressed samples that yield incorrect answers must be strictly excluded from the compression ratio calculation; otherwise, they artificially inflate the metric.
• Fully dynamic continuous compression ratios cause severe optimization failures; models require discrete ratio sets (e.g., 2x, 4x, 8x) to converge effectively.
• Mean-pooling compression consistently outperforms both token-based and last-token extraction methods at equivalent compression rates, contrary to common assumptions about compression tokens.

Evidence (verbatim from paper)

We construct a mixed dataset for evaluation, from four standard reading comprehension benchmarks (filtered under 2, 048 tokens), uniformly sampling 1,000 instances from: HotpotQA, SQuAD, Natural Questions (NQ), and AdversarialQA. We evaluate mainly using 2 metrics: answer accuracy and average compression ratio. For accuracy, we use substring accuracy: a score of 1 is awarded if the exact reference answer appears anywhere within the output, which is more intuitive than F1 and more aligns with human assessment than exact-match. For average compression ratio, it is calculated as the sum of the original context lengths of all the samples which are corrected answered divided by the sum of their compressed lengths. Noteworthy, here we apply a strict validity filter: count only for instances answered correctly.

Citation

@misc{yu2026density,
  title={Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio},
  author={Yijiong Yu et al.},
  year={2026},
  note={arXiv:2603.25926}
}

• arXiv: 2603.25926