Kilt Eval

kilt-eval

KILT: a Benchmark for Knowledge Intensive Language Tasks — Petroni et al. (2020) (arXiv:2009.02252, 2020)

What this evaluates

Evaluates a model's ability to perform knowledge-intensive language tasks by jointly assessing output generation accuracy and evidence retrieval from a fixed Wikipedia snapshot. It measures how well models can produce correct answers while providing verifiable text-span provenance to justify predictions.

Datasets

• KILT — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/facebookresearch/KILT

Metrics

• KILT scores (primary) — range: [0, 1]
  • A joint metric that systematically assesses both the correctness of the generated output and the accuracy of the retrieved provenance (text spans from Wikipedia). It combines downstream task performance with retrieval quality to evaluate evidence-aware reasoning.

Input / output format

Input: Task-specific prompts requiring knowledge-intensive responses, with access to a shared fixed Wikipedia snapshot for retrieval.

Output: Generated answer/response plus associated provenance annotations (text spans from Wikipedia) to justify the prediction.

Scoring recipe

def compute_kilt_score(predictions, gold):
    # predictions and gold contain 'output' and 'provenance' (list of text spans)
    output_acc = exact_match(predictions['output'], gold['output'])
    provenance_acc = exact_match(predictions['provenance'], gold['provenance'])
    # Joint assessment combines both components
    return (output_acc + provenance_acc) / 2

Common pitfalls

• Models without explicit retrieval (e.g., BART, T5) cannot provide provenance, making joint KILT scores inapplicable or requiring special handling.
• Retrieval performance varies significantly across task types; entity-centric tasks favor entity-based retrieval while open QA favors dense passage retrieval.
• Evaluating on the test set requires uploading predictions to the EvalAI challenge rather than local computation.

Evidence (verbatim from paper)

Finally, the KILT scores formulation allows us to systematically assesses the performance for output and provenance jointly (Table 5). We don’t report results for BART and T5 since answers are generated solely from the input with no explicit retrieval and there is no straightforward way to access provenance for each prediction.

Citation

@misc{petroni2020kilt,
  title={KILT: a Benchmark for Knowledge Intensive Language Tasks},
  author={Petroni et al. (2020)},
  year={2020},
  note={arXiv:2009.02252}
}

• arXiv: 2009.02252