Jfin Teb Eval

jfin-teb-eval

JFinTEB: Japanese Financial Text Embedding Benchmark — Suzuki et al. (2026) (arXiv:2604.15882, 2026)

What this evaluates

Evaluates Japanese financial text embedding models across classification, retrieval, and clustering tasks. It probes the models' ability to capture domain-specific semantics, handle regulatory and economic terminology, and generalize to zero-shot financial scenarios.

Datasets

• JFinTEB — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/retarfi/JFinTEB

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Computes the unweighted mean of recall or precision for each class, then averages across all classes. Handles class imbalance by treating each class equally.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Measures ranking quality by discounting the relevance score of each relevant document logarithmically based on its position.
• V-measure — range: [0, 1]
  • Harmonic mean of homogeneity and completeness in clustering evaluation. Homogeneity measures if each cluster contains only members of a single class, while completeness measures if all members of a given class are assigned to the same cluster.

Input / output format

Input: Japanese text instances (sentences, queries, or documents) processed by embedding models to generate fixed-dimensional vector representations.

Output: Predicted class labels (classification), ranked list of retrieved documents (retrieval), or cluster assignments (clustering).

Scoring recipe

def compute_metrics(predictions, gold, task_type):
    if task_type == 'classification':
        return f1_score(gold, predictions, average='macro')
    elif task_type == 'retrieval':
        return ndcg_at_k(gold, predictions, k=10)
    elif task_type == 'clustering':
        return v_measure_score(gold, predictions)

Common pitfalls

• Using validation set scores for final benchmark reporting instead of the held-out test set.
• Including excluded tasks (MultiFin-ja and Industry 33) that showed model-family consistency reversals or validation-test divergence.
• Implementing evaluation logic from scratch instead of using the provided JMTEB codebase, risking protocol inconsistency.

Evidence (verbatim from paper)

Following JMTEB, we adopt standard evaluation protocols for classification (macro-F1), retrieval (NDCG@10), and clustering (V-measure). Our implementation builds directly on the JMTEB codebase with minor modifications to incorporate financial datasets, ensuring consistency and reproducibility across benchmarks. For all tasks, validation sets are used exclusively to select evaluation configurations, while test sets are held out for final reporting.

Citation

@misc{suzuki2026jfin teb,
  title={JFinTEB: Japanese Financial Text Embedding Benchmark},
  author={Suzuki et al. (2026)},
  year={2026},
  note={arXiv:2604.15882}
}

• arXiv: 2604.15882