Loogle Eval

loogle-eval

LooGLE: Can Long-Context Language Models Understand Long Contexts? — Li et al. (2023) (arXiv:2311.04939, 2023)

What this evaluates

Evaluates the ability of language models to comprehend and reason over long documents (up to 32k+ tokens) by testing short and long dependency tasks, including question answering, cloze completion, and summarization.

Datasets

• LooGLE — total 1100; splits: test (1100); repo https://github.com/bigai-nlco/LooGLE

Metrics

• GPT4_score (primary) — range: percent
  • For QA tasks, GPT-4-8k judges semantic equivalence between prediction and ground truth, reported as accuracy. For summarization, GPT-4-8k assigns a score based on multiple generation quality factors.
• Exact Match — range: percent
  • Binary metric where the predicted entity must exactly match the ground truth entity.
• Partial Match — range: percent
  • Fuzzy matching metric allowing approximate or partial overlap between predicted and ground truth entities for cloze tasks.
• Bleu1 — range: percent
  • Standard n-gram precision metric using unigrams to measure lexical overlap between prediction and reference.
• Rouge1 — range: percent
  • Standard n-gram recall metric using unigrams to measure how much of the reference text is covered by the prediction.
• Bert_score — range: [0, 1]
  • Computes cosine similarity between contextual embeddings of prediction and ground truth using a pre-trained BERT model.

Input / output format

Input: Long context document (up to 32k+ tokens) paired with a question, cloze prompt, or summarization instruction. For long dependency tasks, documents may be truncated by concatenating head and tail segments to fit model context windows.

Output: Generated text answer for QA/summarization tasks, or a single entity/token for cloze completion tasks.

Scoring recipe

def score_instance(task, pred, gold):
    if task == 'QA':
        return 1.0 if gpt4_judge_semantic_match(pred, gold) else 0.0
    elif task == 'Summarization':
        return gpt4_judge_quality_score(pred, gold)
    elif task == 'Cloze':
        exact = 1.0 if pred == gold else 0.0
        partial = fuzzy_match_score(pred, gold)
        return {'exact_match': exact, 'partial_match': partial}
    else:
        return {
            'bleu1': compute_bleu(pred, gold, n=1),
            'rouge1': compute_rouge(pred, gold, n=1),
            'bert_score': compute_bert_similarity(pred, gold)
        }

Common pitfalls

• Automatic metrics like BLEU and ROUGE penalize models for generating longer, more fluent answers that are semantically correct but lexically different from the ground truth.
• Truncating long documents by concatenating head and tail segments to fit context windows artificially alters information distribution, potentially masking middle-context degradation or inflating performance.
• GPT-4 evaluator consistency varies across tasks; it may favor specific output styles or struggle with exact entity matching in cloze tasks compared to human judgment.

Evidence (verbatim from paper)

We adopt several automatic evaluation metrics, which can be categorized into two types. Bleu, Rouge, Meteor Score and Bert Score (Li et al., 2023b; Mukherjee & Rahman, 2023) are widely used for generative tasks such as summarization and QA. They evaluate the matching between groundtruth and LLM answers mainly based on n-gram matching and semantic similarity. For Cloze, Exact Match and Partial Match (Sharma et al., 2023; Engelbach et al., 2023) are employed in our evaluation.

Citation

@misc{li2023loogle,
  title={LooGLE: Can Long-Context Language Models Understand Long Contexts?},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2311.04939}
}

• arXiv: 2311.04939