Ettin Arch Comparison Eval

ettin-arch-comparison-eval

Seq vs Seq: An Open Suite of Paired Encoders and Decoders — Weller et al. (2025) (arXiv:2507.11412, 2025)

What this evaluates

Evaluates and compares encoder-only versus decoder-only language models across classification, retrieval, and generative reasoning benchmarks. It specifically probes architectural strengths, the impact of cross-objective continued pre-training, and performance scaling across parameter sizes (XXS to 1B).

Datasets

• GLUE — total ?; splits: test (-1)
• MTEB v2 — total ?; splits: dev (-1)
• Generative/Reasoning Suite (ARC, HellaSwag, LAMBADA, OBQA, SIQA, TQA, WG, WSC) — total ?; splits: test (-1)
• MS MARCO Dev — total ?; splits: dev (-1)

Metrics

• GLUE Avg (primary) — range: percent
  • Macro-average accuracy across SST-2 and MNLI tasks.
• MTEB v2 — range: percent
  • Aggregate score across embedding tasks including CodeSearchNet, MLDR, Clustering, and Retrieval.
• Generative Tasks Avg — range: percent
  • Macro-average accuracy across ARC, HellaSwag, LAMBADA, OpenBookQA, SocialIQA, TriviaQA, Winogrande, and WSC.
• MS MARCO Dev Retrieval — range: percent
  • Retrieval score on the MS MARCO development set.

Input / output format

Input: Text sequences or prompts for classification and retrieval tasks; multiple-choice or open-ended questions for generative tasks.

Output: Class labels for classification; ranked document scores for retrieval; generated text tokens for generative tasks. For cross-objective encoder evaluation on generative tasks, three mask tokens are appended to the sequence and filled iteratively.

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def calculate_macro_avg(task_scores):
    return sum(task_scores) / len(task_scores)

# Example usage for GLUE Avg:
# glues = [calculate_accuracy(sst2_preds, sst2_gold), calculate_accuracy(mnli_preds, mnli_gold)]
# glue_avg = calculate_macro_avg(glues)

Common pitfalls

• Cross-objective evaluation requires specific architectural adaptations, such as appending three mask tokens to encoder sequences for generative tasks instead of standard decoding.
• Size matching is strict (e.g., <1.2B parameters for the 1B category), and continued pre-training with 50B tokens does not bridge the performance gap between encoders and decoders on retrieval/classification tasks.
• Baseline comparisons must account for training recipe differences, as some baselines (e.g., MiniLM) use distillation and have different non-embedding parameter counts.

Evidence (verbatim from paper)

We evaluate on various encoder tasks, including GLUE (Wang et al., 2018), MTEB v2 English (Enevoldsen et al., 2025), MDLR for long context (Chen et al., 2024), and CodeSearchNet for code evaluation (Husain et al., 2019). We use the same evaluation setup as ModernBERT for the evaluation for an equal comparison... We evaluate on a wide range of tasks using the Eleuther AI harness... consolidating tasks used in the Pythia and SmolLM papers including: the ARC Challenge (ARC)... HellaSwag (HS)... LAMBADA (LMB)... OpenBookQA (OBQA)... Social IQA (SIQA)... TriviaQA (TQA)... Winogrande (WG)... and the Winograd Schema Challenge (WSC).

Citation

@misc{weller2025seqvsseq,
  title={Seq vs Seq: An Open Suite of Paired Encoders and Decoders},
  author={Weller et al. (2025)},
  year={2025},
  note={arXiv:2507.11412}
}

• arXiv: 2507.11412