Lllm Paper Filtering Eval

lllm-paper-filtering-eval

LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models — Kostikova et al. (2025) (arXiv:2505.19240, 2025)

What this evaluates

Evaluates the ability of LLMs to accurately classify academic papers as discussing LLM limitations and to extract supporting evidence from abstracts. It measures alignment with human expert annotations using ordinal rating agreement and span-level extraction metrics.

Datasets

• ACL Anthology & arXiv (crawled 2022-2025) — total ?; splits: test (-1); repo https://github.com/a-kostikova/LLLMs-Survey

Metrics

• weighted-cohens-kappa (primary) — range: [-1, 1]
  • Quadratic weighted Cohen's Kappa measuring agreement between LLM predictions and human expert ratings on an ordinal 0-5 scale for LLM limitation discussions.
• pairwise-f1 — range: [0, 1]
  • F1 score computed between model-extracted evidence spans and human-annotated evidence spans in abstracts.

Input / output format

Input: Paper abstracts (and full text for evidence extraction), accompanied by prompt templates defining limitation categories and rating scales.

Output: Ordinal limitation rating (0-5) and extracted evidence spans/sentences from the abstract.

Scoring recipe

import statsmodels.stats.inter_rater as irr

def compute_kappa(pred_ratings, gold_ratings):
    return irr.cohens_kappa(pred_ratings, gold_ratings, weights='quadratic')

def compute_f1(pred_spans, gold_spans):
    pred_set, gold_set = set(pred_spans), set(gold_spans)
    tp = len(pred_set & gold_set)
    fp, fn = len(pred_set - gold_set), len(gold_set - pred_set)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Human annotators and models frequently confuse adjacent ordinal categories (e.g., 2 vs 3, 3 vs 4) due to the subjective nature of limitation depth.
• Evidence extraction F1 is highly sensitive to span granularity; models tend to select 1-2 key sentences while humans extract longer arguments, causing systematic precision/recall trade-offs.

Evidence (verbatim from paper)

We report quadratic weighted Cohen’s Kappa for limitation ratings and pairwise F1 for evidence extraction, measured between each annotator and the model for the best-performing prompts.

Citation

@misc{kostikova2025lllms,
  title={LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models},
  author={Kostikova et al. (2025)},
  year={2025},
  note={arXiv:2505.19240}
}

• arXiv: 2505.19240