Factuality Score

factuality_score

StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluation — Kashyap et al. (2025) (arXiv:2507.21340, 2025)

What this evaluates

Evaluates the quality of synthetically generated natural language reports derived from tabular data. It probes factual grounding, narrative coherence, hallucination, and the precise preservation of numerical and temporal information from the source table.

Datasets

• StructText-Synthetic — total 71539; splits: test (71539); repo https://github.com/ibm/struct-text

Metrics

• factuality_score (primary) — range: 1-5
  • 1-5 rubric scored by an LLM-as-judge. 5 indicates fully correct with all claims supported by source data; 1 indicates fundamentally incorrect with most claims contradicted.
• hallucination_score — range: 1-5
  • 1-5 rubric scored by an LLM-as-judge. 5 indicates no hallucinations (all content grounded or attributed); 1 indicates heavy hallucination with numerous invented details.
• coherence_score — range: 1-5
  • 1-5 rubric scored by an LLM-as-judge. 5 indicates seamless narrative flow with natural transitions; 1 indicates incoherent text with random jumps.
• numeric_precision — range: [0, 1]
  • Ratio of correctly extracted numeric values in the generated text to the total number of numeric values extracted from the text. Uses a 0.1% relative error tolerance for matching.
• numeric_recall — range: [0, 1]
  • Ratio of correctly extracted numeric values in the generated text to the total number of numeric values in the ground truth source row. Uses a 0.1% relative error tolerance for matching.
• temporal_precision — range: [0, 1]
  • Ratio of correctly extracted temporal values in the generated text to the total number of temporal values extracted from the text. Uses LLM extraction with SUTime fallback for normalization.
• temporal_recall — range: [0, 1]
  • Ratio of correctly extracted temporal values in the generated text to the total number of temporal values in the ground truth source row. Uses LLM extraction with SUTime fallback for normalization.

Input / output format

Input: A generated natural language report string and the corresponding source tabular row (ground truth) containing the original numeric, temporal, and categorical values.

Output: Per instance: LLM-as-judge scores (1-5) for factuality, hallucination, and coherence; and precision/recall values (floats in [0,1]) for numeric and temporal extraction accuracy.

Scoring recipe

def llm_judge(text, source, dim):
    prompt = f'Rubric for {dim}: {RUBRIC}\nText: {text}\nSource: {source}\nScore 1-5:'
    return llm.generate(prompt, return_rationale=True)

def extract_match(text, source, vtype):
    gt = parse_values(source, vtype) # CoreNLP/SUTime
    pred = parse_values(text, vtype)
    matches = sum(1 for p in pred if any(abs(p-g)/g <= 0.001 for g in gt))
    return matches/len(pred), matches/len(gt)

Common pitfalls

• LLM-as-judge scores require self-consistency checks and explicit rationales to mitigate judge hallucination.
• Numeric matching uses a 0.1% relative error tolerance, so exact string matching will incorrectly penalize rounded values.
• Temporal parsing must handle diverse natural language formats (e.g., 'Q4 2022' vs 'fourth quarter of 2022'), requiring context-aware LLM extraction or SUTime fallback.

Evidence (verbatim from paper)

We adopt the LLM as a judge paradigm to evaluate the quality of our generated reports along dimensions which are difficult to measure through other heuristic or objective measures. Specifically, we use LLMs to evaluate three critical aspects of the generated text: hallucination detection... coherence... and factuality... For each measure, we design 5-point rubrics... For numeric values, we combine Stanford’s Core NLP’s NER parser with regular expression patterns... apply a 0.1% relative error tolerance... determine the precision and recall of numeric values in the generated text.

Citation

@misc{kashyap2025structtext,
  title={StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluation},
  author={Kashyap et al. (2025)},
  year={2025},
  note={arXiv:2507.21340}
}

• arXiv: 2507.21340