Robust Mean Relative Error

robust-mean-relative-error

Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models — Bordt et al. (2024) (arXiv:2404.06209, 2024)

What this evaluates

Evaluates a model's ability to forecast future values in a financial time-series dataset given historical observations. It measures prediction accuracy using a capped relative error to prevent extreme outliers from dominating the score.

Datasets

• MSCI World stock index — total ?; splits: test (-1)

Metrics

• robust mean relative error (primary) — range: other: [0, 0.02]
  • Average over days t=20 to T of the minimum of the absolute relative error |(y_hat_t - y_t)/y_t| and 0.02. This caps the penalty for large errors at 2% per day.

Input / output format

Input: System prompt specifying the forecasting task, followed by the price levels of the stock index for the previous 20 days.

Output: A single predicted price value for the current day.

Scoring recipe

total_error = 0.0
for t in range(20, T):
    rel_err = abs(pred[t] - actual[t]) / abs(actual[t])
    total_error += min(rel_err, 0.02)
metric_value = total_error / (T - 20)

Common pitfalls

• The metric caps individual daily relative errors at 0.02, so large prediction mistakes do not linearly penalize the overall score.
• The evaluation summation starts at t=20, implying a specific indexing or warm-up period that must be respected when aligning predictions with ground truth.

Evidence (verbatim from paper)

For each year, we compute the robust mean relative error as

$$ \frac {1}{T - 2 0} \sum_ {t = 2 0} ^ {T} \min \left{\frac {\left| \hat {y} _ {t} - y _ {t} \right|}{\left| y _ {t} \right|}, 0. 0 2 \right}. \tag {1}

We use a simple system prompt.

System: "You are an expert financial market analyst and stock trader and your task is to predict the development of the MSCI World stock index in 2022. You are given the price level of the stock index in the previous 20 days and predict the price of the stock index on the current day."

Citation

@misc{bordt2024elephants,
  title={Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models},
  author={Bordt et al. (2024)},
  year={2024},
  note={arXiv:2404.06209}
}

• arXiv: 2404.06209