Fastfact Eval

fastfact-eval

FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs — Wan et al. (2025) (arXiv:2510.12839, 2025)

What this evaluates

Evaluates the long-form factuality of LLM-generated responses by extracting claims, verifying them against evidence, and comparing the system's factuality scores against human annotations.

Datasets

• FaStfact-Bench — total 400; splits: test (400); repo https://github.com/Yingjia-Wan/FastFact

Metrics

• F₁@K′ (primary) — range: [0, 1]
  • F1 score computed at a threshold K′, measuring the overlap between predicted and ground-truth claim counts/scores.

Input / output format

Input: Long-form QA pairs (question + generated response) with ground-truth human annotations for claim counts and factuality.

Output: Predicted factuality score (F₁@K′), extracted claim count, and total token cost per sample.

Scoring recipe

def compute_metrics(predictions, gold):
    pred_claims = extract_claims(predictions['response'])
    gt_claims = gold['claim_count']
    f1_at_k_prime = f1_score(pred_claims, gt_claims, threshold=K_prime)
    delta_k = abs(pred_claims - gt_claims)
    delta_f1 = abs(f1_at_k_prime - gold['f1_score'])
    token_cost = count_tokens(predictions['response'])
    return f1_at_k_prime, delta_k, delta_f1, token_cost

Common pitfalls

• Using sentence-level chunking (stride=1) causes over-decomposition, inflating claim counts and token costs without improving accuracy.
• Token cost is reported as the primary efficiency proxy, which may not reflect actual inference latency or wall-clock time.
• Alignment metrics ($|\Delta K|$ and $|\Delta F_{1}@K^{\prime}|$) measure absolute deviation, potentially masking systematic bias in claim extraction.

Evidence (verbatim from paper)

By utilizing the ground-truth annotations from FaStfact-Bench, we run FaStfact and several baselines in parallel using the same underlying LLM as the extractor and verifier to compare their (1) evaluation reliability, proxied by the alignment with human judgment average $|\Delta K|$ and $|\Delta$$F_{1}@K^{\prime}$ $|$, and (2) practical efficiency, proxied by total token cost per sample.

Citation

@misc{wan2025fastfact,
  title={FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs},
  author={Wan et al. (2025)},
  year={2025},
  note={arXiv:2510.12839}
}

• arXiv: 2510.12839