Freshretailnet 50k Eval

freshretailnet-50k-eval

FreshRetailNet-50K: A Stockout-Annotated Censored Demand Dataset for Latent Demand Recovery and Forecasting in Fresh Retail — Wang et al. (2025) (arXiv:2505.16319, 2025)

What this evaluates

This benchmark evaluates models' ability to recover latent demand during stockout periods in perishable retail. It probes whether algorithms can disentangle true consumption patterns from supply-induced censoring using hourly temporal data and contextual covariates. Success is measured by prediction accuracy, bias mitigation, and the decoupling of recovered demand from stockout ratios.

Datasets

• FreshRetailNet-50K — total 50000; splits: (unstated); repo https://github.com/Dingdong-Inc/frn-50k-baseline

Metrics

• WAPE (primary) — range: percent
  • Weighted Absolute Percentage Error: sum(|d_t - y_t|) / sum(y_t). Measures magnitude accuracy of recovered demand.
• WPE — range: percent
  • Weighted Percentage Error: sum(d_t - y_t) / sum(y_t). Measures bias direction (over/under-prediction).
• rho_DS — range: [-1, 1]
  • Decoupling Score: weighted Pearson correlation between stockout ratios and recovered demand across store-product pairs, where weights are proportional to mean sales.

Input / output format

Input: Hourly time-series sales data with contextual covariates (weather, promotions, holidays) for specific store-product pairs.

Output: Predicted true demand values (y_t) for each hourly time step t.

Scoring recipe

import numpy as np
def compute_metrics(preds, gold, stockout_ratios, mean_sales):
    wape = np.sum(np.abs(gold - preds)) / np.sum(gold)
    wpe = np.sum(gold - preds) / np.sum(gold)
    weights = mean_sales / np.sum(mean_sales)
    pearson_corr = np.corrcoef(stockout_ratios, preds)[0, 1]
    rho_ds = np.sum(weights * pearson_corr)
    return wape, wpe, rho_ds

Common pitfalls

• Evaluating directly on observed sales during stockouts instead of recovering latent demand, which violates the MNAR simulation protocol.
• Focusing exclusively on WAPE while ignoring WPE, thereby missing systematic bias that causes inventory cascades.
• Treating all store-product pairs equally in the Decoupling Score without applying the mean-sales weighting (w_i), which skews results toward high-volume items.

Evidence (verbatim from paper)

Model performance is quantified through Weighted Absolute Percentage Error(WAPE) and Weighted Percentage Error(WPE): $$ \mathrm {W A P E} = \frac {\sum_ {t} \left| d _ {t} - y _ {t} \right|}{\sum_ {t} y _ {t}} \quad (\text {m a g n i t u d e a c c u r a c y}) \tag {4} $$ $$ \mathrm {W P E} = \frac {\sum_ {t} \left(d _ {t} - y _ {t}\right)}{\sum_ {t} y _ {t}} \quad (\text {b i a s d i r e c t i o n}) \tag {5} $$

Citation

@misc{wang2025freshretailnet,
  title={FreshRetailNet-50K: A Stockout-Annotated Censored Demand Dataset for Latent Demand Recovery and Forecasting in Fresh Retail},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.16319}
}

• arXiv: 2505.16319