Candlestick Forecasting Eval

candlestick-forecasting-eval

Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting — Hu et al. (2026) (arXiv:2604.12659, 2026)

What this evaluates

Evaluates Vision-Language Models' ability to analyze multi-scale candlestick charts (daily and weekly) and predict 30-day forward stock returns. It probes their capacity for visual technical analysis, trend recognition, and regression-based financial forecasting without relying on textual market data.

Datasets

• Multi-Scale Candlestick Stock Return Benchmark — total ?; splits: test (-1)

Metrics

• IC (Information Coefficient) (primary) — range: other
  • Pearson linear correlation between predicted returns and actual 30-day forward returns. Mean IC reflects average predictive capability; ICIR = mean IC / std(IC).
• Rank IC (Spearman Correlation) — range: other
  • Spearman’s rank correlation between predicted and actual returns, providing robustness against outliers.
• Classification Metrics (Accuracy, Precision, Recall, F1) — range: percent
  • Derived from a confusion matrix where predictions and actual returns are binarized at 0. Accuracy = (TP+TN)/Total, Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = harmonic mean of Precision and Recall.

Input / output format

Input: Two candlestick charts (Daily and Weekly) for the same stock on the same date, accompanied by a structured prompt defining the model's role, a multidimensional technical analysis framework (candlestick patterns, moving averages, volume, inflection points), and few-shot examples.

Output: A single numerical score in the range [-0.5, 1.0], rounded to three decimal places, enclosed within tags, with no additional text.

Scoring recipe

def compute_metrics(preds, actuals):
    ic = pearsonr(preds, actuals)[0]
    rank_ic = spearmanr(preds, actuals)[0]
    icir = ic / np.std(ic) if np.std(ic) > 0 else 0.0
    y_true = (actuals > 0).astype(int)
    y_pred = (preds > 0).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'IC': ic, 'Rank IC': rank_ic, 'ICIR': icir, 'Accuracy': accuracy, 'Precision': precision, 'Recall': recall, 'F1': f1}

Common pitfalls

• Models may exhibit strong directional bias despite instructions to avoid it, leading to inflated accuracy but poor IC.
• Confusion between daily and weekly chart signals can cause misclassification of pullbacks versus trend reversals.
• Thresholding continuous predictions at 0 for classification metrics may mask calibration issues in the [-0.5, 1.0] range.

Evidence (verbatim from paper)

The Information Coefficient (IC) measures the linear correlation between predicted values and actual returns. The mean IC reflects the average level of predictive capability, while the median IC indicates predictive stability. The Information Coefficient Ratio (ICIR), calculated as the mean IC divided by the standard deviation of IC, measures risk-adjusted predictive capability.

Citation

@misc{hu2026candlestick,
  title={Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting},
  author={Hu et al. (2026)},
  year={2026},
  note={arXiv:2604.12659}
}

• arXiv: 2604.12659