Fair Summm Eval

fair-summm-eval

Fair Abstractive Summarization of Diverse Perspectives — Zhang et al. (2023) (arXiv:2311.07884, 2023)

What this evaluates

Evaluates the fairness of abstractive summarization by measuring distributional alignment across social attributes (e.g., sentiment, gender, party). It quantifies how well generated summaries preserve the proportion of diverse perspectives present in the source text, penalizing underrepresentation of minority viewpoints.

Datasets

• PERSPECTIVESUMM — total ?; splits: Claritin (-1), US Election (-1), Yelp (-1), Amazon (-1), SupremeCourt (-1), IQ2 (-1); repo https://github.com/psunlpgroup/FairSumm

Metrics

• Binary Unfair Rate (BUR) (primary) — range: percent
  • Measures the proportion of generated summaries that underrepresent a target social attribute or perspective compared to the source distribution. Lower scores indicate better fairness.
• Unfair Error Rate (UER) — range: percent
  • Quantifies the magnitude of distributional deviation between the summary and source across social attributes. Lower scores indicate better alignment.

Input / output format

Input: Source text (e.g., product reviews, tweets, political articles) with optional instruction prompts.

Output: Abstractive summary generated by the LLM.

Scoring recipe

def compute_bur_uer(source, summary, values, tau=0.8):
    p_x = count_distribution(source, values)
    p_y = count_distribution(summary, values)
    ratios = [p_y[v] / p_x[v] for v in values if p_x[v] > 0]
    unfair_count = sum(1 for r in ratios if r < 1 - tau)
    BUR = (unfair_count / len(source_samples)) * 100
    UER = mean(abs(ratios - 1)) * 100
    return BUR, UER

Common pitfalls

• BUR and UER are reference-free metrics, so they do not require human-written reference summaries for evaluation.
• Lower scores are better for both BUR and UER (indicated by ↓ in tables).
• Metric values are sensitive to the softmax temperature used in the underlying scoring model (e.g., BARTScore); the paper aligns them using temperature=0.1.

Evidence (verbatim from paper)

proposing four reference-free metrics—Binary Unfair Rate (BUR), Unfair Error Rate (UER), and variants—to quantify underrepresentation of perspectives.

Citation

@misc{zhang2023fairsumm,
  title={Fair Abstractive Summarization of Diverse Perspectives},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2311.07884}
}

• arXiv: 2311.07884