qhjqhj00 qrecc-attribution-fluency-eval
Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models — Aksitov et al. (2023) (arXiv:2302.05578, 2023)
What this evaluates
Evaluates the tradeoff between response fluency and factual attribution in retrieval-augmented conversational LLMs. It measures how well models generate coherent, context-aware responses while correctly grounding answers in provided evidence or dialog history.
Datasets
- QReCC — total ?; splits: test (-1)
Metrics
Auto-AIS(primary) — range: [0, 1]- Automated scoring metric for attribution accuracy. Exact formula not provided in the experimental setup; likely measures alignment between generated response and provided evidence/facts.
Auto-SSA— range: [0, 1]- Automated scoring metric for response fluency and coherence. Exact formula not provided in the experimental setup; validated against human ratings in a pilot study.
Input / output format
Input: Conversational turns formatted as 'turn_index parent_index speaker_id utterance [eot]', optionally prefixed with 'Instructions:' and 'Fact:' sections containing retrieved, golden, absent, or non-evidence text.
Output: Model-generated text completion, parsed by stopping at the next '[eot]' token.
Scoring recipe
def evaluate(predictions, gold, metric_type):
if metric_type == 'Auto-AIS':
return compute_attribution_alignment(predictions, gold_evidence)
elif metric_type == 'Auto-SSA':
return compute_fluency_coherence(predictions, dialog_history)
# Note: Exact implementation details for Auto-AIS and Auto-SSA are defined in the full paper, not the setup section.
Common pitfalls
- Stopping generation at the wrong [eot] token can truncate valid responses or include prompt artifacts.
- Using non-evidence or absent facts without adjusting the prompt structure may artificially deflate attribution scores.
- Temperature 0.0 vs 0.7 significantly impacts fluency scores, making cross-setting comparisons sensitive to sampling parameters.
Evidence (verbatim from paper)
To better understand the impact of various components, we further adjust structure of the prompts along several dimensions: “instructions” could be either present or absent, similarly, dialog history could be also present or absent, and the provided evidence could be: golden, retrieved, absent, non-evidence... We arrive into a “full grid” of experiments by running the prompting setups described above in 6 settings each: 3 different model sizes (8B, 62B and 540B) and 2 sampling temperatures (0.0 and 0.7). As was mentioned in the previous section, conducting the full grid of experiments with human raters is unrealistic. Instead, after confirming that the auto-metrics are well-aligned with the human ratings from the pilot, we apply Auto-AIS and Auto-SSA to the full grid of experiments for further analysis.
Citation
@misc{aksitov2023characterizing,
title={Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models},
author={Aksitov et al. (2023)},
year={2023},
note={arXiv:2302.05578}
}
- arXiv: 2302.05578