Tfrb Eval

tfrb-eval

TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems — Ahamed et al. (2026) (arXiv:2604.05364, 2026)

What this evaluates

Evaluates the causal reasoning and forecasting accuracy of LLMs and time-series models on multi-domain time-series data. It probes whether step-by-step reasoning and external event context improve numerical predictions or introduce narrative bias, particularly in stochastic versus pattern-rich domains.

Datasets

• TFRBench — total ?; splits: test (-1)

Metrics

• MASE (primary) — range: other
  • Mean Absolute Scaled Error: MASE = MAE_model / MAE_naive, where the naive forecast uses the previous time step. Lower values indicate better forecasting accuracy.
• LLM-as-Judge Score — range: [1, 5]
  • LLM-evaluated reasoning quality scored on a 1-5 Likert scale across four dimensions: domain relevance, forecasting correctness, event relevance, and logic consistency. The average score across dimensions and samples is reported.
• Overall Success Rate — range: percent
  • Percentage of test samples where the model's MASE is strictly less than 1.0.

Input / output format

Input: Historical time-series context window (potentially augmented with external event/holiday data). Models receive system prompts enforcing role adherence and are asked to either output a direct numerical forecast or generate a step-by-step reasoning trace before forecasting.

Output: Either a single numerical prediction, or a structured reasoning trace (analyzing trends, seasonality, cross-channel dependencies, and event predictions) followed by a final numerical forecast.

Scoring recipe

def compute_mase(predictions, gold):
    mae_model = mean(abs(predictions - gold))
    mae_naive = mean(abs(gold[1:] - gold[:-1]))
    return mae_model / mae_naive

def compute_success_rate(predictions, gold):
    mases = [compute_mase(p, g) for p, g in zip(predictions, gold)]
    return sum(1 for m in mases if m < 1.0) / len(mases) * 100

def compute_judge_score(reasoning_traces):
    # Gemini-3-Pro evaluates each trace on 4 dimensions (1-5 scale)
    # Returns average score across dimensions and samples
    return llm_judge.evaluate(traces, dimensions=[
        'domain_relevance', 'forecasting_correctness',
        'event_relevance', 'logic_consistency'
    ])

Common pitfalls

• In stochastic domains like Finance, step-by-step reasoning can introduce 'narrative bias' where models hallucinate causal links, actually degrading forecasting accuracy compared to direct prediction.
• Event forecasting with external search acts as a distractor in high-entropy technical workloads (e.g., Web/CloudOps), causing significant error spikes when models over-rely on irrelevant external news.
• Evaluating only final numerical accuracy without auditing reasoning traces misses critical failures in causal attribution and logic consistency.

Evidence (verbatim from paper)

Figure 4 reports MASE across three distinct experimental settings for the corresponding models. We report the mean and standard deviation $(\mathrm{mean}_{\mathrm{std}})$ across three runs for statistical significance. Default parameters are used for reasoning models, while the "w/ Reasoning" setting enforces an external CoT trace. Detailed results are provided in App. J.

Citation

@misc{ahamed2026tfrbench,
  title={TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems},
  author={Ahamed et al. (2026)},
  year={2026},
  note={arXiv:2604.05364}
}

• arXiv: 2604.05364