Fine Grained Ie Eval

fine-grained-ie-eval

Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction — Gao et al. (2023) (arXiv:2310.05092, 2023)

What this evaluates

This benchmark evaluates large language models' capacity for fine-grained information extraction under augmented instructions. It specifically probes two generalization capabilities: adapting to unseen information types and handling unfamiliar task forms. Performance is measured using standard extraction metrics to compare encoder-decoder and decoder-only architectures.

Datasets

• Fine-grained Information Extraction Benchmark — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall calculated over extracted spans or tuples for each information type and task form.

Input / output format

Input: Input text paired with augmented instructions (task descriptions, extraction rules, output formats, and examples) for in-context learning or fine-tuning.

Output: Structured extractions (entities, triggers, arguments, or ASTE tuples) conforming to the specified output format.

Scoring recipe

def compute_f1(preds, gold):
    pred_set = set(preds)
    gold_set = set(gold)
    tp = len(pred_set & gold_set)
    prec = tp / len(pred_set) if pred_set else 0
    rec = tp / len(gold_set) if gold_set else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Assuming model performance scales monotonically with parameter count; results show architecture and instruction design matter more.
• Expecting encoder-decoder models to adapt well to new task forms; they generalize poorly to unfamiliar structures compared to decoder-only models using in-context learning.

Evidence (verbatim from paper)

Specifically, the T5 models with 3B and 11B parameters achieved F1 scores of 82.45 and 78.70 respectively in the entity extraction task. These scores significantly surpass the highest F1 score (64.25) achieved by ChatGPT in the decoder-only category.

Citation

@misc{gao2023benchmarking,
  title={Benchmarking Large Language Models with Augmented Instructions for Fine-grained Information Extraction},
  author={Gao et al. (2023)},
  year={2023},
  note={arXiv:2310.05092}
}

• arXiv: 2310.05092