Odqa Compression Eval

odqa-compression-eval

Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models — Kim (2025) (arXiv:2512.08943, 2025)

What this evaluates

Evaluates the ability of abstractive compression models to preserve factual correctness and answer strings when processing noisy retrieved documents in open-domain question answering. It measures how well compressed summaries retain key information to support downstream answer generation while reducing context length and inference latency.

Datasets

• Natural Questions (NQ) — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)
• PopQA — total ?; splits: test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • Character-level exact match between the generated answer and the reference answer. Returns 1 if identical, 0 otherwise.
• F1 score — range: [0, 1]
  • Token-level F1 score balancing precision and recall of the predicted answer against the reference, minimizing omissions.
• Compression Ratio (CR) — range: [0, 1]
  • Ratio of compressed summary length to original document length, measuring summarization efficiency.
• Preserving Answer string Ratio (PAR) — range: [0, 1]
  • Fraction of queries where the exact answer string is preserved verbatim in the compressed summary output.
• Inference Time — range: seconds
  • Wall-clock time in seconds taken by the downstream language model to process the compressed input and generate a response.

Input / output format

Input: Query $q$, top-5 retrieved documents $D$, and a compression instruction $I_c$.

Output: Abstractive compressed summary $S$ (used as context for a downstream LLM to generate the final answer).

Scoring recipe

def compute_metrics(final_answers, gold_answers, summaries, original_docs):
    em = sum(1 for a, g in zip(final_answers, gold_answers) if a.strip() == g.strip()) / len(final_answers)
    f1 = mean([f1_score(g, a) for a, g in zip(final_answers, gold_answers)])
    cr = mean([len(s) / len(d) for s, d in zip(summaries, original_docs)])
    par = sum(1 for s, g in zip(summaries, gold_answers) if g in s) / len(summaries)
    return {'EM': em, 'F1': f1, 'CR': cr, 'PAR': par}

Common pitfalls

• CR is defined as output/input length ratio here, whereas many compression papers report it as input/output or percentage reduction.
• PAR is evaluated on the compressed summary text itself, not on the final answer generated by the downstream LLM.
• Inference time only captures the downstream LLM's generation latency, not the compressor's processing time.

Evidence (verbatim from paper)

Specifically, EM measures how precisely the system's answer matches the reference answer at the character level, while the F1 score balances precision and recall, evaluating the accuracy of identified answers and minimizing omissions. CR evaluates how efficiently the compressor summarizes the information essential to answer the query.

Citation

@misc{kim2025noiserobust,
  title={Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models},
  author={Kim (2025)},
  year={2025},
  note={arXiv:2512.08943}
}

• arXiv: 2512.08943