Context Is Key Eval

context-is-key-eval

Context is Key: A Benchmark for Forecasting with Essential Textual Information — Williams et al. (2024) (arXiv:2410.18959, 2024)

What this evaluates

Evaluates a model's ability to integrate essential natural language context with numerical time series data to produce accurate forecasts. It probes multimodal reasoning, constraint satisfaction, and the capacity to leverage textual information for improving time series prediction.

Datasets

• CiK — total ?; splits: test (-1); repo https://github.com/ServiceNow/context-is-key-forecasting

Metrics

• RCRPS (primary) — range: other
  • Region of Interest Continuous Ranked Probability Score. Computed over a specified region of the time series. Forecasts with errors exceeding 500% of the ground truth range are clipped to 5 to prevent disproportionate skewing of aggregate scores. Lower values indicate better performance.

Input / output format

Input: A numerical time series history paired with a natural language context string describing task-specific constraints or information.

Output: A structured forecast containing predictions for all required future timestamps.

Scoring recipe

def compute_rcrpss(predictions, gold, context_range):
    crps = compute_crps(predictions, gold)
    if abs(crps) > 5 * context_range:
        crps = 5.0
    return crps

# For each instance, average 25 independent forecasts
instance_score = mean([compute_rcrpss(f, g, r) for f in forecasts])
# Aggregate across tasks using cluster weighting
aggregate = weighted_mean(instance_scores, cluster_weights)

Common pitfalls

• Models often ignore the provided natural language context, performing similarly to purely numerical baselines.
• Significant forecasting failures (errors >500% of ground truth range) can disproportionately dominate aggregate scores if not clipped.
• Instruction tuning can degrade autoregressive forecasting performance for certain LLM architectures.

Evidence (verbatim from paper)

A method is considered to outperform another on a task if its average RCRPS is lower on said task. ... we clip the RCRPS of such instances to 5 to avoid them disproportionately skewing the aggregate score.

Citation

@misc{williams2024contextiskey,
  title={Context is Key: A Benchmark for Forecasting with Essential Textual Information},
  author={Williams et al. (2024)},
  year={2024},
  note={arXiv:2410.18959}
}

• arXiv: 2410.18959