Information Sufficiency

information-sufficiency

FLARE: Task-agnostic embedding model evaluation through a normalization process — Jingzhou Jiang et al. (2026) (arXiv:2604.17344, 2026)

What this evaluates

Evaluates the quality of text embedding models in a task-agnostic manner by estimating information sufficiency via normalizing flows. It predicts how well an embedding model will perform on downstream tasks without requiring task-specific labels or fine-tuning.

Datasets

• Apt-eval — total ?; splits: test (-1)
• GT-FintechLab — total ?; splits: test (-1)
• BhashaBench-Finance — total ?; splits: test (-1)
• AIR-Bench-Finance — total ?; splits: test (-1)
• LIMIT — total ?; splits: test (-1)
• ArXiv-Abstracts 2025 — total ?; splits: test (-1); HF almanach/arxiv_abstracts_2025
• Augmented STS-B — total ?; splits: test (-1); HF maiammar/Augmented_stsb_multi_mt
• LivNLP-STS — total ?; splits: test (-1)
• Philosophical-STS — total ?; splits: test (-1); HF johnnyboycurtis/Philosophical-STS-Text-Pairs
• Clustered-FunPang Medical — total ?; splits: test (-1); HF mukulb/clustered_FUNPANG_dataset_with_groups
• Reasoning-Clustering — total ?; splits: test (-1); HF Ibisbill/Clustering_deduplicated_reasoning

Metrics

• Spearman's ρ (primary) — range: [-1, 1]
  • Spearman's rank correlation coefficient between the predicted Information Sufficiency (IS) scores and the ground-truth supervised benchmark scores (F1 macro, nDCG@10, Spearman correlation, or V-measure depending on task).

Input / output format

Input: High-dimensional text embeddings (d=384–4096) generated by candidate models on 11 held-out datasets. For retrieval tasks, only passage corpus embeddings are used.

Output: A single scalar Information Sufficiency (IS) score per model per dataset, computed as the median of per-pair IS scores derived from normalizing flow log-likelihoods.

Scoring recipe

# 1. Compute IS score for each model on each dataset
is_scores = []
for model in models:
    for dataset in datasets:
        embeddings = model.encode(dataset)
        is_score = compute_is_score(embeddings) # via normalizing flows
        is_scores.append(is_score)

# 2. Compute ground-truth supervised metric for each model/dataset
gt_scores = []
for model in models:
    for dataset in datasets:
        gt_scores.append(get_supervised_metric(model, dataset)) # F1, nDCG@10, etc.

# 3. Compute ranking alignment
spearman_rho = spearmanr(is_scores, gt_scores).correlation

Common pitfalls

• Using kernel or GMM density estimators in high-dimensional spaces (d ≥ 3584) causes vanishingly sparse estimates and systematic ranking inversions.
• Assuming embedding spaces should be uniform or penalizing anisotropy fails because high-quality semantic spaces are inherently anisotropic.
• Averaging per-pair IS scores is sensitive to heavy-tailed distributions; median aggregation is required to avoid noise from geometric mismatches.

Evidence (verbatim from paper)

We assess the reliability of our unsupervised Information Sufficiency (IS) metric by measuring its alignment with ground-truth supervised rankings. Ground-truth performance is established using standard MTEB metrics: F1 macro for classification, nDCG@10 for retrieval, Spearman correlation for STS, and V-measure for clustering. We quantify ranking alignment using Spearman’s rank correlation ($\rho$) and Pearson correlation ($r$) between the predicted IS scores and the supervised metrics.

Citation

@misc{jiang2026flare,
  title={FLARE: Task-agnostic embedding model evaluation through a normalization process},
  author={Jingzhou Jiang et al. (2026)},
  year={2026},
  note={arXiv:2604.17344}
}

• arXiv: 2604.17344