Long Form Scientific Summarization Eval

long-form-scientific-summarization-eval

What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization — Griffin Adams et al. (2023) (arXiv:2305.07615, 2023)

What this evaluates

Evaluates long-form scientific summarization models on their ability to generate relevant and faithful abstracts across clinical, chemical, and biomedical domains. It probes how calibration set construction and candidate selection strategies affect model performance on standard relevance and faithfulness metrics.

Datasets

• Scientific Summarization Datasets — total ?; splits: Clinical (-1), Chemical (-1), Biomedical (-1); repo https://github.com/griff4692/calibrating-summaries

Metrics

• Rouge-1 F1 (primary) — range: [0, 1]
  • F1 score of unigram (1-gram) overlap between generated summary and reference. Computed using standard ROUGE implementation.
• Rouge-2 F1 — range: [0, 1]
  • F1 score of bigram (2-gram) overlap between generated summary and reference.
• BERTScore F1 (vs reference) — range: [0, 1]
  • F1 score computed from cosine similarity between contextual embeddings of generated tokens and reference tokens, averaged across tokens.
• FactScore — range: [0, 1]
  • LLM-based faithfulness metric that verifies factual claims in the generated summary against the source text, reporting the proportion of verified claims.
• BARTScore — range: other
  • Encoder-decoder based metric that computes the log-likelihood ratio of the generated summary given the reference versus the reference given the generated summary.
• BERTScore F1 (vs source) — range: [0, 1]
  • F1 score computed from cosine similarity between generated summary tokens and source paper tokens.
• Rel_Agg — range: other
  • Normalized aggregate relevance improvement over the fine-tuned baseline, calculated as standard deviation improvements across datasets.
• Faith_Agg — range: other
  • Normalized aggregate faithfulness improvement over the fine-tuned baseline, calculated as standard deviation improvements across datasets.

Input / output format

Input: Source scientific paper text

Output: Generated long-form summary (abstract)

Scoring recipe

def score_summary(generated, reference, source):
    r1 = rouge_score(generated, reference, rouge_types=['rouge1']).fmeasure
    r2 = rouge_score(generated, reference, rouge_types=['rouge2']).fmeasure
    bs_ref = bertscore(generated, reference, model_type='google/pegasus-pubmed').f1.mean()
    fs = factscore(generated, source)
    bs = bartscore(generated, reference).score.mean()
    bs_src = bertscore(generated, source, model_type='google/pegasus-pubmed').f1.mean()
    return {'Rouge-1 F1': r1, 'Rouge-2 F1': r2, 'BERTScore F1 (ref)': bs_ref,
            'FactScore': fs, 'BARTScore': bs, 'BERTScore F1 (src)': bs_src}

Common pitfalls

• Clinical references are highly noisy and often unsupported by the source text, which artificially deflates relevance metrics compared to chemical/biomedical domains.
• BARTScore was trained on PubMed, causing a domain mismatch that scores chemical summaries significantly lower than biomedical ones.
• Relevance and faithfulness exhibit a strong inverse relationship; optimizing for one typically degrades the other, requiring joint calibration strategies.
• High extractiveness in generated summaries can inflate relevance scores while masking factual errors, leading to metric-reality gaps.

Evidence (verbatim from paper)

R1, R2, and BS-Ref stand for Rouge-1/2 F1 and BERTScore F1 vis-a-vis reference, respectively. Fact., Bart., and BS-Src stand for FactScore, BARTScore, and BERTScore F1 vis-a-vis the source. Metrics defined in §4.1 and 4.2.

Citation

@misc{adams2023calibrationsets,
  title={What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization},
  author={Griffin Adams et al. (2023)},
  year={2023},
  note={arXiv:2305.07615}
}

• arXiv: 2305.07615