Longllmlingua Eval

longllmlingua-eval

LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — Jiang et al. (2023) (arXiv:2310.06839, 2023)

What this evaluates

Evaluates the effectiveness of a question-aware prompt compression framework on long-context LLM tasks. It measures how well compressed prompts preserve key information and answer accuracy across multi-document QA, summarization, and code completion scenarios.

Datasets

• NaturalQuestions (Liu et al., 2023) — total ?; splits: test (-1)
• LongBench (Bai et al., 2023) — total ?; splits: test (-1)
• ZeroSCROLLS (Shaham et al., 2023) — total ?; splits: validation (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered questions where the model's output matches the ground truth answer.
• LongBench official metrics — range: varies
  • Task-specific metrics provided by the LongBench benchmark (e.g., exact match, ROUGE, F1) evaluated using the benchmark's official scripts.
• ZeroSCROLLS official metrics — range: varies
  • Task-specific metrics provided by the ZeroSCROLLS benchmark evaluated using the benchmark's official scripts.

Input / output format

Input: Long-context prompts containing a user question and multiple retrieved documents (e.g., 20 documents for NaturalQuestions), processed through a question-aware compression pipeline before being fed to a target LLM (GPT-3.5-Turbo-0613 or LongChat-13B-16k).

Output: Model-generated response (e.g., extracted answer, summary, or code completion) produced via greedy decoding (temperature=0).

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_text(pred) == normalize_text(gold):
            correct += 1
    return correct / len(predictions)

# For LongBench & ZeroSCROLLS:
# Run official benchmark evaluation scripts provided with each dataset
# to compute task-specific metrics (e.g., F1, ROUGE, exact match)

Common pitfalls

• Using temperature > 0 or sampling instead of greedy decoding (temp=0) breaks reproducibility and inflates variance.
• Failing to account for the five specific ground-truth document position settings (1st, 5th, 10th, 15th, 20th) in NaturalQuestions can artificially inflate or deflate accuracy.
• Substituting custom evaluation code for the official LongBench and ZeroSCROLLS scripts yields non-comparable results.

Evidence (verbatim from paper)

We use NaturalQuestions for the multi-document QA task, and use LongBench and ZeroSCROLLS for general long context scenarios. ... Following Liu et al. (2023), we use accuracy as the evaluation metric. ... We use the metrics and scripts provided along with the benchmark for evaluation.

Citation

@misc{jiang2023longllmlingua,
  title={LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression},
  author={Jiang et al. (2023)},
  year={2023},
  note={arXiv:2310.06839}
}

• arXiv: 2310.06839