Swimba Standard Bench Eval

swimba-standard-bench-eval

Swimba: Switch Mamba Model Scales State Space Models — Du et al. (2026) (arXiv:2603.06938, 2026)

What this evaluates

Evaluates language understanding, reasoning, and knowledge recall capabilities of a Switch Mamba model on standard multiple-choice benchmarks. It also measures inference efficiency (throughput, latency, FLOPs) to assess the computational trade-offs of the parameter-space mixture-of-experts design.

Datasets

• BoolQ — total ?; splits: test (-1)
• OpenBookQA — total ?; splits: test (-1)
• RTE — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• PIQA — total ?; splits: test (-1)
• WinoGrande — total ?; splits: test (-1)
• HellaSwag — total ?; splits: test (-1)
• ARC-Challenge — total ?; splits: test (-1)
• ARC-Easy — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • The fraction of examples where the correct choice has the highest total log-likelihood.
• Normalized accuracy — range: [0, 1]
  • The fraction of examples where the correct choice has the highest length-normalized log-likelihood, computed by dividing each choice log-likelihood by its answer length in bytes.

Input / output format

Input: Multiple-choice questions or prompts with candidate answers, processed via LM-Evaluation-Harness.

Output: Log-likelihood scores for each candidate answer choice.

Scoring recipe

def compute_accuracy(lls, gold_idx):
    return int(np.argmax(lls) == gold_idx)

def compute_normalized_accuracy(lls, lengths, gold_idx):
    norm_lls = [ll / length for ll, length in zip(lls, lengths)]
    return int(np.argmax(norm_lls) == gold_idx)

# Aggregate over dataset
acc_scores = [compute_accuracy(pred['lls'], gold) for pred, gold in zip(predictions, golds)]
norm_acc_scores = [compute_normalized_accuracy(pred['lls'], pred['lengths'], gold) for pred, gold in zip(predictions, golds)]
return sum(acc_scores) / len(golds), sum(norm_acc_scores) / len(golds)

Common pitfalls

• Using greedy decoding or generation-based accuracy instead of log-likelihood evaluation, which LM-Evaluation-Harness requires for multiple-choice tasks.
• Ignoring length normalization when computing normalized accuracy, leading to inflated scores for longer answer choices.
• Reporting single-run results without standard deviations, whereas the paper explicitly reports subscripts for standard deviation across runs.

Evidence (verbatim from paper)

For performance with LM-Evaluation-Harness, we report accuracy, the fraction of examples where the correct choice has the highest total log-likelihood; and normalized accuracy, the fraction of examples where the correct choice has the highest length-normalized log-likelihood, computed by dividing each choice log-likelihood by its answer length in bytes

Citation

@misc{du2026swimba,
  title={Swimba: Switch Mamba Model Scales State Space Models},
  author={Du et al. (2026)},
  year={2026},
  note={arXiv:2603.06938}
}

• arXiv: 2603.06938