Nifty Stock Movement Eval

nifty-stock-movement-eval

NIFTY Financial News Headlines Dataset — Saqur et al. (2024) (arXiv:2405.09747, 2024)

What this evaluates

Evaluates LLMs on predicting short-term stock price movements based on financial news headlines and market context. It probes the model's ability to extract directional sentiment and predictive signals from textual financial data for downstream forecasting tasks.

Datasets

• NIFTY Financial News Headlines Dataset — total ?; splits: test (-1)

Metrics

• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• Accuracy — range: [0, 1]
  • Ratio of correctly predicted instances to the total number of instances.
• Precision — range: [0, 1]
  • Ratio of true positive predictions to the total number of positive predictions.
• Recall — range: [0, 1]
  • Ratio of true positive predictions to the total number of actual positives.

Input / output format

Input: Financial news headline with market context, formatted as a consistent LM query $x_q$.

Output: Classification label indicating stock movement direction (e.g., up/down or specific movement class).

Scoring recipe

def compute_metrics(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == g == 1)
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(predictions, gold_labels) if p == g == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = (tp + tn) / len(gold_labels)
    return {'F1 Score': f1, 'Accuracy': accuracy, 'Precision': precision, 'Recall': recall}

Common pitfalls

• Stock movement prediction is inherently noisy; baseline accuracies often hover near random chance (e.g., 0.24–0.45 in this paper), so modest gains should be interpreted cautiously.
• The evaluation strictly relies on the Flare benchmark's standardized query format ($x_q$); deviating from this prompt structure breaks compatibility with the reported results.
• The paper only reports test split performance; no validation split or early stopping protocol is described, which may lead to overfitting if not controlled.

Evidence (verbatim from paper)

Table 5: Performance of a single baseline expert Llama-2-7b-chat with 4 variants (LoRA SFT adapters) on the NIFTY Stock Price Movement Prediction Task (test split).

Metrics ↑ F1 Score Accuracy Precision Recall

Citation

@misc{saqur2024nifty,
  title={NIFTY Financial News Headlines Dataset},
  author={Saqur et al. (2024)},
  year={2024},
  note={arXiv:2405.09747}
}

• arXiv: 2405.09747