Encoder Adaptation Eval

encoder-adaptation-eval

Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks — Suganthan et al. (2025) (arXiv:2503.02656, 2025)

What this evaluates

This evaluation protocol assesses the capability of decoder-based language models adapted into encoder-only architectures to perform diverse downstream tasks, including text classification, scoring, and information retrieval. It specifically probes how architectural modifications like bidirectional attention, pooling strategies, and dropout affect performance on standard benchmarks.

Datasets

• GLUE — total ?; splits: train (-1), eval (-1); HF glue
• SuperGLUE — total ?; splits: train (-1), eval (-1); HF super_glue
• MS MARCO — total 530000; splits: train (530000), dev (7000)

Metrics

• GLUE score — range: [0, 1]
  • Macro-average of task-specific metrics (accuracy, F1, or Pearson correlation) across all GLUE tasks.
• SuperGLUE score — range: [0, 1]
  • Macro-average of task-specific metrics across all SuperGLUE tasks (excluding RECORD).
• MRR@10 (primary) — range: [0, 1]
  • Mean Reciprocal Rank of the first relevant document among the top 10 retrieved results. Calculated as the average of 1/rank for each query.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Measures ranking quality by weighting relevance scores by their logarithmic position, normalized against the ideal ranking.

Input / output format

Input: For classification tasks: single sentences or sentence pairs formatted per task. For ranking tasks: a query paired with a candidate passage (or list of passages) to be scored for relevance.

Output: For classification: predicted class label. For ranking: relevance score or rank order for each candidate passage.

Scoring recipe

def compute_mrr_at_10(relevance_scores, top_k=10):
    for i, rel in enumerate(relevance_scores[:top_k]):
        if rel == 1:
            return 1.0 / (i + 1)
    return 0.0

def compute_ndcg_at_10(relevance_scores, top_k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores[:top_k]))
    ideal_rels = sorted(relevance_scores, reverse=True)[:top_k]
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_rels))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Using causal attention masking by default, which significantly degrades performance on encoder tasks compared to bidirectional masking.
• Applying inappropriate pooling strategies (e.g., attention pooling) without considering the attention masking type; last-token pooling is preferred when causal masking is used.
• Including the RECORD task from SuperGLUE, which is incompatible with encoder-only architectures and must be excluded.

Evidence (verbatim from paper)

Following the setup in RankT5*(Zhuang et al., [2022])*, our evaluation focuses on the top $1000$ retrieved documents using MRR@10 and NDCG@10 as metrics, while our training uses a sample of $36$ documents ($1$ positive plus sampled $35$ negatives) per query.

Citation

@misc{suganthan2025adapting,
  title={Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks},
  author={Suganthan et al. (2025)},
  year={2025},
  note={arXiv:2503.02656}
}

• arXiv: 2503.02656