Nsw Epnews Eval

nsw-epnews-eval

NSW-EPNews: A News-Augmented Benchmark for Electricity Price Forecasting with LLMs — Bi et al. (2025) (arXiv:2506.11050, 2025)

What this evaluates

This benchmark evaluates the ability of traditional time-series models and large language models to forecast half-hourly electricity prices in New South Wales, Australia. It specifically probes how well models integrate multimodal inputs (historical prices, weather, and market news) and tests for numerical reasoning capabilities, hallucination resistance, and strict output formatting compliance in a high-stakes forecasting task.

Datasets

• NSW-EPNews — total ?; splits: full (-1), recent_50pct (-1), recent_30pct (-1), recent_10pct (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error; the average of absolute differences between predicted and actual half-hourly electricity prices over the 48-step forecast horizon.
• RMSE — range: other
  • Root Mean Squared Error; the square root of the average of squared differences between predicted and actual prices.
• MAPE — range: percent
  • Mean Absolute Percentage Error; the average of absolute percentage differences between predicted and actual prices, expressed as a percentage.

Input / output format

Input: For traditional models: a feature vector combining the last 10 half-hour prices, a 50-dimensional TF-IDF representation of the day's news, and the day's minimum and maximum temperatures. For LLMs: structured prompts containing historical price/load data, background news/context, and instructions to predict the next 48 half-hour steps.

Output: A sequence of exactly 48 numerical values representing the forecasted half-hourly prices, formatted as comma-separated numbers without spaces, quotes, newlines, or markdown.

Scoring recipe

def score(predictions, gold):
    preds = extract_numbers(predictions)
    if len(preds) != 48:
        return {'format_violation': True, 'MAE': None}
    if is_degenerate(preds):
        return {'hallucination_type': 'degenerate_copy', 'MAE': None}
    mae = mean(abs(preds - gold))
    rmse = sqrt(mean((preds - gold)**2))
    mape = mean(abs((preds - gold) / gold)) * 100
    return {'MAE': mae, 'RMSE': rmse, 'MAPE': mape}

Common pitfalls

• LLMs frequently violate strict output formats by adding conversational text, markdown, or code blocks instead of raw comma-separated numbers.
• Models often exhibit degenerate behaviors such as echoing the last historical price, repeating a single value 48 times, or applying a trivial fixed offset to the input sequence.

Evidence (verbatim from paper)

Then LLMs' predicted price sequence will be extracted and scored against the true prices using MAE, MSE, RMSE and MAPE.

Citation

@misc{bi2025nswepnews,
  title={NSW-EPNews: A News-Augmented Benchmark for Electricity Price Forecasting with LLMs},
  author={Bi et al. (2025)},
  year={2025},
  note={arXiv:2506.11050}
}

• arXiv: 2506.11050