Financial Sts Eval

financial-sts-eval

Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives — Liu et al. (2024) (arXiv:2403.14341, 2024)

What this evaluates

Evaluates a model's ability to detect subtle semantic shifts between pairs of financial narratives by ranking similar pairs higher than dissimilar ones. It probes nuanced understanding of financial language, including intensified sentiment, elaborated details, plan realization, and emerging situations.

Datasets

• LLM-augmented FinSTS — total ?; splits: train (-1), test (-1)
• Human-annotated FinSTS — total ?; splits: test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. It measures the probability that a randomly chosen positive (similar) pair receives a higher similarity score than a randomly chosen negative (dissimilar) pair.

Input / output format

Input: Pairs of financial narrative sentences.

Output: Cosine similarity score between the sentence embeddings.

Scoring recipe

def compute_auc(predictions, gold_labels):
    # predictions: list of cosine similarity scores
    # gold_labels: list of binary labels (1=similar, 0=dissimilar)
    from sklearn.metrics import roc_auc_score
    auc = roc_auc_score(gold_labels, predictions)
    return auc

Common pitfalls

• AUC evaluates ranking quality rather than direct similarity score accuracy, so models can score high on AUC while having poorly calibrated similarity scores.
• The LLM-augmented test set is drawn from the same generation process as the training set, so high AUC there reflects memorization/domain alignment rather than true generalization; the human-annotated test set is the proper out-of-distribution benchmark.
• Cosine similarity is used as the raw output for ranking, not a normalized [0,1] similarity metric, which can affect threshold-dependent evaluations if misinterpreted.

Evidence (verbatim from paper)

We use Area Under the ROC Curve (AUC) as the evaluation metric to assess the quality of identified semantic simiarlity between a pair of financial narratives. A high AUC means that a model ranks the positive (similar) pairs higher than the negative (dissimilar) pairs consistently.

Citation

@misc{liu2024financialsts,
  title={Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2403.14341}
}

• arXiv: 2403.14341