Openforesight Eval

openforesight-eval

Scaling Open-Ended Reasoning to Predict the Future — Chandak et al. (2025) (arXiv:2512.25070, 2025)

What this evaluates

Evaluates language models' ability to make probabilistic forecasts on open-ended, future-uncertain questions derived from global news. It probes both prediction accuracy and calibration, testing whether models can generalize forecasting skills across diverse sources and time horizons without leaking future information.

Datasets

• OpenForesight Test Set — total 302; splits: test (302)
• FutureX — total 86; splits: test (86)
• SimpleQA — total ?; splits: test (-1)
• MMLU-Pro — total ?; splits: test (-1)
• GPQA-Diamond — total ?; splits: test (-1)

Metrics

• Brier Score (primary) — range: [0, 1]
  • Mean squared error between predicted probability and binary outcome (0 or 1). Lower values indicate better accuracy and calibration.
• Accuracy — range: [0, 1]
  • Fraction of correctly predicted outcomes out of total questions.
• Arbitrage & Frequentist Consistency Metrics — range: percent
  • Metrics proposed by Paleka et al. (2025b) to measure long-term prediction consistency on binary questions up to 2028. Reported as percentage improvement over baseline.

Input / output format

Input: Open-ended forecasting questions derived from daily global news articles, typically asking for a binary or multiple-choice resolution outcome with a specified date range.

Output: Probabilistic predictions (confidence scores between 0 and 1) for each question, optionally abstaining with 'I do not know' if confidence falls below a threshold (e.g., < 0.1).

Scoring recipe

def compute_brier_score(predictions, outcomes):
    return sum((p - o) ** 2 for p, o in zip(predictions, outcomes)) / len(predictions)

def compute_accuracy(predictions, outcomes):
    return sum(1 for p, o in zip(predictions, outcomes) if p == o) / len(predictions)

# Brier score is the primary evaluation metric; accuracy is reported alongside it.

Common pitfalls

• Future information leakage: Models may inadvertently access articles published after the question's resolution date if retrieval is not strictly offline or filtered.
• Source distribution bias: Using news sources for testing that overlap with training data can inflate performance due to stylistic or topical familiarity rather than genuine forecasting ability.
• Late reporting: News outlets may report answers retroactively; failing to filter for earliest resolution dates introduces leakage.

Evidence (verbatim from paper)

On the Brier score (Y axis), the primary metric recommended for forecasting (Tetlock & Gardner, 2016), as it measures both accuracy and calibration, OpenForecaster8B outperforms even GPT OSS 120B. Our improvements are not merely from calibration, the predictions also become more accurate (X axis), beating Qwen3 235B, but are a bit behind others.

Citation

@misc{chandak2025scalingopenendedreasoning,
  title={Scaling Open-Ended Reasoning to Predict the Future},
  author={Chandak et al. (2025)},
  year={2025},
  note={arXiv:2512.25070}
}

• arXiv: 2512.25070