Tsver Eval

tsver-eval

TSVer: A Benchmark for Fact Verification Against Time-Series Evidence — Strong et al. (2025) (arXiv:2511.01101, 2025)

What this evaluates

This benchmark evaluates an AI system's ability to perform fact verification using time-series evidence. It probes multi-timeframe temporal reasoning, cross-series numerical analysis, and the generation of factually consistent justifications aligned with human annotations.

Datasets

• TSVer — total 287; splits: test (-1), development (-1); repo https://github.com/marekstrong/TSVer

Metrics

• Accuracy (primary) — range: percent
  • Proportion of correctly predicted verdicts (SUPPORTED or REFUTED) out of the total number of evaluation instances.
• TSCS — range: percent
  • TSCS = (1/N) Σ (F1_i * J_bar_i), where F1_i measures the dataset-level selection accuracy and J_bar_i is the average Jaccard Index over matched time ranges.
• Ev²R — range: percent
  • F1 score computed from precision and recall over atomic facts extracted from model-generated justifications compared against reference justifications.

Input / output format

Input: A natural language claim paired with retrieved time-series data formatted as Markdown-style tables (or raw numerical records).

Output: A binary verdict (SUPPORTED or REFUTED), a textual justification explaining the verdict, and the set of retrieved time-series datasets with their corresponding time ranges.

Scoring recipe

def compute_tscs(predictions, gold):
    total = 0.0
    for pred, g in zip(predictions, gold):
        ds_f1 = f1_score(pred.datasets, g.datasets)
        jaccs = []
        for gt_ds in g.datasets:
            if gt_ds in pred.datasets:
                pred_r = pred.time_ranges[gt_ds]
                gt_r = g.time_ranges[gt_ds]
                jaccs.append(jaccard_index(pred_r, gt_r))
        avg_j = sum(jaccs) / len(jaccs) if jaccs else 0.0
        total += ds_f1 * avg_j
    return (total / len(predictions)) * 100

Common pitfalls

• Models frequently over-retrieve time series or time spans, exceeding context windows and causing inference failures.
• Surface-level lexical metrics like METEOR fail to correlate with human judgments of factual adequacy in justifications.
• BPE tokenization of floating-point numbers yields inefficient representations, degrading numerical reasoning on continuous data.

Evidence (verbatim from paper)

We propose the Time Series Coverage Score (TSCS), a metric that jointly captures the accuracy of both time series selection and temporal coverage. TSCS combines a dataset-level F1 score with a temporal Jaccard Index to evaluate the quality of each retrieval instance.

Citation

@misc{strong2025tsver,
  title={TSVer: A Benchmark for Fact Verification Against Time-Series Evidence},
  author={Strong et al. (2025)},
  year={2025},
  note={arXiv:2511.01101}
}

• arXiv: 2511.01101