Futurex Eval

futurex-eval

FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction — Zhiyuan Zeng et al. (2025) (arXiv:2508.11987, 2025)

What this evaluates

Evaluates LLM agents' ability to forecast real-world future events under uncertainty. It probes reasoning depth, tool-use/search capability, and temporal validity by requiring models to answer dynamic, live-updated questions before event resolution.

Datasets

• FutureX — total ?; splits: test (-1)

Metrics

• overall_score (primary) — range: [0, 1]
  • Weighted average of tier-level accuracy: 0.1 × Acc(Level 1) + 0.2 × Acc(Level 2) + 0.3 × Acc(Level 3) + 0.4 × Acc(Level 4). Tier accuracy is the proportion of correctly predicted events within that difficulty tier.

Input / output format

Input: Event description or question collected from live web sources, labeled with a difficulty tier (Level 1–4). Prompts may be single/multiple-choice or open-ended.

Output: Predicted outcome or selected answer option for the specified future event.

Scoring recipe

def compute_overall_score(predictions, gold, tier_labels):
    tier_acc = {}
    for tier in ['Level 1', 'Level 2', 'Level 3', 'Level 4']:
        mask = [t == tier for t in tier_labels]
        if sum(mask) == 0: continue
        correct = sum(1 for p, g in zip(predictions, gold) if p == g and mask[i])
        tier_acc[tier] = correct / sum(mask)
    weights = {'Level 1': 0.1, 'Level 2': 0.2, 'Level 3': 0.3, 'Level 4': 0.4}
    return sum(weights[t] * tier_acc.get(t, 0.0) for t in weights)

Common pitfalls

• Predictions must be made prospectively before event resolution; post-hoc answers constitute data leakage and invalidate the live benchmark premise.
• Level 1/2 events are multiple-choice while Level 3/4 are open-ended, requiring different evaluation strategies (exact match vs. semantic/LLM-judge scoring).
• Human baseline comparisons used a different question subset than the automated model tests, so reported performance gaps are approximate and not directly comparable.

Evidence (verbatim from paper)

As for the overall score, we combine scores from the 4 difficulty tiers (see Table 3) using weights of 10%, 20%, 30%, and 40%, respectively, with heavier weights assigned to the more challenging tiers.

Citation

@misc{zeng2025futurex,
  title={FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction},
  author={Zhiyuan Zeng et al. (2025)},
  year={2025},
  note={arXiv:2508.11987}
}

• arXiv: 2508.11987