# Bidirlm Eval

> Evaluates the capability of adapted causal LLMs to function as bidirectional encoders across text, vision, and audio modalities, measuring performance on downstream fine-tuning tasks and zero-shot/linear-probing embedding benchmarks. Use when the user wants to benchmark on MTEB v2 (English & Multilingual), MIRACL, CodeSearchNet, MNLI, XNLI, PAWS-X, MathShepherd, CodeComplexity, PAN-X, POS, Seahorse, MIEB lite, MAEB beta, Beaver, Safe, Aegis, e-SNLI-VE, BoolQ, or asks about evaluating this task. Reports classification accuracy / nDCG@10 / regression metrics.

- Skill: `qhjqhj00/bidirlm-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/bidirlm-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/bidirlm-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/bidirlm-eval

---


# bidirlm-eval

> BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs — Boizard et al. (2026) (arXiv:2604.02045, 2026)

## What this evaluates

Evaluates the capability of adapted causal LLMs to function as bidirectional encoders across text, vision, and audio modalities, measuring performance on downstream fine-tuning tasks and zero-shot/linear-probing embedding benchmarks.

## Datasets

- **MTEB v2 (English & Multilingual)** — total ?; splits: test (-1)
- **MIRACL** — total ?; splits: test (-1)
- **CodeSearchNet** — total ?; splits: test (-1)
- **MNLI** — total ?; splits: test (-1)
- **XNLI** — total ?; splits: test (-1)
- **PAWS-X** — total ?; splits: test (-1)
- **MathShepherd** — total ?; splits: test (-1)
- **CodeComplexity** — total ?; splits: test (-1)
- **PAN-X** — total ?; splits: test (-1)
- **POS** — total ?; splits: test (-1)
- **Seahorse** — total ?; splits: test (-1)
- **MIEB lite** — total ?; splits: test (-1)
- **MAEB beta** — total ?; splits: test (-1)
- **Beaver** — total ?; splits: test (-1)
- **Safe** — total ?; splits: test (-1)
- **Aegis** — total ?; splits: test (-1)
- **e-SNLI-VE** — total ?; splits: test (-1)
- **BoolQ** — total ?; splits: test (-1)

## Metrics

- `classification accuracy / nDCG@10 / regression metrics` **(primary)** — range: [0, 1] | percent
  - Task-dependent: accuracy for classification/entailment/comprehension tasks; nDCG@10 or MRR for information retrieval; Pearson/Spearman correlation for sequence regression. Exact metric names are not explicitly stated in the text but follow standard conventions for each task category.

## Input / output format

**Input**: Task-dependent: text pairs/triples for IR/SC/TC/SR; image-text or audio-text pairs for cross-modal benchmarks; full sequences for fine-tuning downstream tasks.

**Output**: Task-dependent: class labels for classification/entailment; relevance scores/rankings for IR; continuous values for regression; dense vectors for embedding benchmarks.

## Scoring recipe

```python
def compute_metric(predictions, gold, task_type):
    if task_type in ['classification', 'entailment', 'comprehension']:
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'ir':
        return ndcg_at_k(predictions, gold, k=10)
    elif task_type == 'regression':
        return pearsonr(predictions, gold)
    elif task_type == 'embedding':
        return cosine_similarity_score(predictions, gold)
    else:
        raise ValueError('Unknown task type')
```

## Common pitfalls

- Confusing adaptation corpora (FineWeb-Edu, KaLM-embedding, Omni-Contrastive) with evaluation datasets, which would cause severe data leakage and inflated scores.
- Mixing up the two evaluation paradigms: fine-tuning uses full-parameter adaptation on downstream tasks, while embedding evaluation uses zero-shot or linear probing on off-the-shelf representations.
- Overlooking modality alignment requirements for cross-modal benchmarks (MIEB lite, MAEB beta), which expect specific image-text or audio-text pairings that text-only adapters cannot natively satisfy without explicit multimodal adaptation.

## Evidence (verbatim from paper)

> To reflect the current usage landscape, we assess encoder performance across diverse representation tasks under two distinct paradigms: 1. Fine-tuning evaluation: We apply full-parameter adaptation for downstream tasks spanning the XTREME benchmark and four specific task categories: Information Retrieval (IR) via MIRACL and CodeSearchNet; Sequence Classification (SC) via MNLI, XNLI, PAWS-X, MathShepherd, and CodeComplexity; Token Classification (TC) via PAN-X and POS; and Sequence Regression (SR) via Seahorse. 2. Embedding evaluation: We assess off-the-shelf embedding performance via zero-shot and linear probing on MTEB-style benchmarks. Text evaluation uses English and Multilingual MTEB v2, while cross-modal capabilities rely on MIEB lite and MAEB beta.

## Citation

```bibtex
@misc{boizard2026bidirlm,
  title={BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs},
  author={Boizard et al. (2026)},
  year={2026},
  note={arXiv:2604.02045}
}
```

- arXiv: 2604.02045