Financial Phrase Bank Eval

financial-phrase-bank-eval

Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts — Malo et al. (2013) (arXiv:1307.5336, 2013)

What this evaluates

This benchmark probes a model's ability to classify financial news sentences or phrases into positive, neutral, or negative semantic orientations. It specifically tests domain-specific sentiment analysis by evaluating how well models capture contextual cues, economic concepts, and directional event expectations in financial texts.

Datasets

• Financial PhraseBank — total ?; splits: 100% agreement (2259), >75% agreement (3448), >66% agreement (4211), >50% agreement (4840)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Single sentences or phrases extracted from financial news articles.

Output: One of three class labels: Positive, Neutral, or Negative.

Scoring recipe

def compute_metrics(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    accuracy = correct / len(gold)
    precisions, recalls = [], []
    for label in ['Positive', 'Neutral', 'Negative']:
        tp = sum(1 for p, g in zip(predictions, gold) if p == label and g == label)
        fp = sum(1 for p, g in zip(predictions, gold) if p == label and g != label)
        fn = sum(1 for p, g in zip(predictions, gold) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        precisions.append(prec)
        recalls.append(rec)
    avg_prec = sum(precisions) / 3
    avg_rec = sum(recalls) / 3
    f1 = 2 * (avg_prec * avg_rec) / (avg_prec + avg_rec) if (avg_prec + avg_rec) > 0 else 0
    return accuracy, f1

Common pitfalls

• The dataset is partitioned by inter-annotator agreement strength rather than a fixed train/validation/test split, requiring 10-fold cross-validation for evaluation.
• The class distribution is highly imbalanced, with Neutral sentences comprising roughly 60% of the data, which can artificially inflate accuracy if not monitored.
• Baseline models rely on different external lexicons (e.g., MPQA vs. Loughran & McDonald), making direct performance comparisons sensitive to lexicon quality rather than just algorithmic complexity.

Evidence (verbatim from paper)

Tables 4 and 5 show performance of the models on the four reference datasets defined based on the phrase bank with different degrees of inter-annotator agreement. The results reported for the algorithms with a machine learning component (i.e. MPQA and the two variants of LPS) are computed using 10-fold cross-validation. ... The accuracy levels achieved by the better performing LPS algorithm ranged from 0.828 to 0.951 on sentences with 100% agreement, and between 0.792 and 0.945 on the sentences with more than 75% agreement. Also the F1 score, which is defined as the harmonic mean of precision and recall was very high for LPS in comparison to the corresponding ranges for MPQA, W-Loughran and W-MPQA.

Citation

@misc{maloe2013gooddebt,
  title={Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts},
  author={Malo et al. (2013)},
  year={2013},
  note={arXiv:1307.5336}
}

• arXiv: 1307.5336