Korfinsts Eval

korfinsts-eval

NMIXX: Domain-Adapted Neural Embeddings for Cross-Lingual eXploration of Finance — Lee et al. (2025) (arXiv:2507.09601, 2025)

What this evaluates

Evaluates the ability of cross-lingual embedding models to capture nuanced financial semantics and terminology in low-resource Korean text, specifically measuring how well they align with human judgments of sentence similarity in specialized financial contexts.

Datasets

• KorFinSTS — total 1921; splits: test (1921)

Metrics

• Spearman’s ρ (primary) — range: [-1, 1]
  • Spearman's rank correlation coefficient between the model-predicted similarity scores and the human-annotated similarity scores on a 0–5 scale.

Input / output format

Input: Pairs of Korean financial sentences (e.g., from news, investment reports, regulatory disclosures, or legal texts).

Output: A continuous similarity score, typically on a 0–5 scale, representing the semantic similarity between the two sentences.

Scoring recipe

def compute_spearman_rho(predictions, gold):
    from scipy.stats import spearmanr
    rho, _ = spearmanr(predictions, gold)
    return rho

Common pitfalls

• Using general-domain STS benchmarks instead of domain-specific financial ones, which fails to capture specialized terminology and nuanced financial framing.
• Relying on directly translated English datasets, which often lose cultural context and terminological precision, leading to significant performance degradation in low-resource settings.
• Ignoring the 0–5 human annotation scale and treating the task as a binary classification or regression without proper rank-based evaluation.

Evidence (verbatim from paper)

STS is the problem of measuring the semantic similarity of texts. ... The final benchmark consists of 1,921 high-quality sentence pairs. ... Similarity Score Distribution (0-5 Scale) ... Mean Score 0.59 ... The model achieves +0.22 Spearman’s ρ gains on KorFinSTS (a new Korean financial STS benchmark) and outperforms general-purpose baselines by the largest margin, demonstrating superior capture of financial semantics in low-resource languages.

Citation

@misc{lee2025nmixx,
  title={NMIXX: Domain-Adapted Neural Embeddings for Cross-Lingual eXploration of Finance},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2507.09601}
}

• arXiv: 2507.09601