Bidirlm Eval

bidirlm-eval

BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs — Boizard et al. (2026) (arXiv:2604.02045, 2026)

What this evaluates

Evaluates the capability of adapted causal LLMs to function as bidirectional encoders across text, vision, and audio modalities, measuring performance on downstream fine-tuning tasks and zero-shot/linear-probing embedding benchmarks.

Datasets

• MTEB v2 (English & Multilingual) — total ?; splits: test (-1)
• MIRACL — total ?; splits: test (-1)
• CodeSearchNet — total ?; splits: test (-1)
• MNLI — total ?; splits: test (-1)
• XNLI — total ?; splits: test (-1)
• PAWS-X — total ?; splits: test (-1)
• MathShepherd — total ?; splits: test (-1)
• CodeComplexity — total ?; splits: test (-1)
• PAN-X — total ?; splits: test (-1)
• POS — total ?; splits: test (-1)
• Seahorse — total ?; splits: test (-1)
• MIEB lite — total ?; splits: test (-1)
• MAEB beta — total ?; splits: test (-1)
• Beaver — total ?; splits: test (-1)
• Safe — total ?; splits: test (-1)
• Aegis — total ?; splits: test (-1)
• e-SNLI-VE — total ?; splits: test (-1)
• BoolQ — total ?; splits: test (-1)

Metrics

• classification accuracy / nDCG@10 / regression metrics (primary) — range: [0, 1] | percent
  • Task-dependent: accuracy for classification/entailment/comprehension tasks; nDCG@10 or MRR for information retrieval; Pearson/Spearman correlation for sequence regression. Exact metric names are not explicitly stated in the text but follow standard conventions for each task category.

Input / output format

Input: Task-dependent: text pairs/triples for IR/SC/TC/SR; image-text or audio-text pairs for cross-modal benchmarks; full sequences for fine-tuning downstream tasks.

Output: Task-dependent: class labels for classification/entailment; relevance scores/rankings for IR; continuous values for regression; dense vectors for embedding benchmarks.

Scoring recipe

def compute_metric(predictions, gold, task_type):
    if task_type in ['classification', 'entailment', 'comprehension']:
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'ir':
        return ndcg_at_k(predictions, gold, k=10)
    elif task_type == 'regression':
        return pearsonr(predictions, gold)
    elif task_type == 'embedding':
        return cosine_similarity_score(predictions, gold)
    else:
        raise ValueError('Unknown task type')

Common pitfalls

• Confusing adaptation corpora (FineWeb-Edu, KaLM-embedding, Omni-Contrastive) with evaluation datasets, which would cause severe data leakage and inflated scores.
• Mixing up the two evaluation paradigms: fine-tuning uses full-parameter adaptation on downstream tasks, while embedding evaluation uses zero-shot or linear probing on off-the-shelf representations.
• Overlooking modality alignment requirements for cross-modal benchmarks (MIEB lite, MAEB beta), which expect specific image-text or audio-text pairings that text-only adapters cannot natively satisfy without explicit multimodal adaptation.

Evidence (verbatim from paper)

To reflect the current usage landscape, we assess encoder performance across diverse representation tasks under two distinct paradigms: 1. Fine-tuning evaluation: We apply full-parameter adaptation for downstream tasks spanning the XTREME benchmark and four specific task categories: Information Retrieval (IR) via MIRACL and CodeSearchNet; Sequence Classification (SC) via MNLI, XNLI, PAWS-X, MathShepherd, and CodeComplexity; Token Classification (TC) via PAN-X and POS; and Sequence Regression (SR) via Seahorse. 2. Embedding evaluation: We assess off-the-shelf embedding performance via zero-shot and linear probing on MTEB-style benchmarks. Text evaluation uses English and Multilingual MTEB v2, while cross-modal capabilities rely on MIEB lite and MAEB beta.

Citation

@misc{boizard2026bidirlm,
  title={BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs},
  author={Boizard et al. (2026)},
  year={2026},
  note={arXiv:2604.02045}
}

• arXiv: 2604.02045