Securerouter Eval

securerouter-eval

SecureRouter: Encrypted Routing for Efficient Secure Inference — Zhang et al. (2026) (arXiv:2604.15499, 2026)

What this evaluates

This evaluation probes the accuracy and inference efficiency of an encrypted routing framework for secure Transformer inference. It measures how well a cost-aware router dynamically selects smaller MPC-optimized models from a pool to balance privacy-preserving computation costs with task-specific accuracy requirements.

Datasets

• GLUE — total ?; splits: test (-1); HF glue

Metrics

• Accuracy — range: [0, 1]
  • Percentage of correctly predicted labels on classification tasks (MNLI, RTE, SST-2, QNLI).
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for binary/multi-class tasks (MRPC, QQP).
• Matthews correlation coefficient — range: [-1, 1]
  • Correlation coefficient between true and predicted labels, robust to class imbalance (CoLA).
• Pearson/Spearman correlation — range: [-1, 1]
  • Average of Pearson and Spearman rank correlations for regression tasks (STS-B).
• Inference Speed-up (primary) — range: other
  • Ratio of baseline BERT-Large MPC latency to the weighted sum of selected expert latencies plus router overhead: (N × C_Large) / Σ(C_selected + C_router).

Input / output format

Input: Encrypted text sequences representing GLUE benchmark tasks (classification or regression).

Output: Predicted class labels or regression scores, along with a routing decision indicating which expert model (Tiny, Base, or Large) processes the sample.

Scoring recipe

For each task, compute the designated metric (accuracy, F1, MCC, or avg correlation) against gold labels.
Compute speed-up as: speedup = (N * C_Large) / sum(C_selected[i] + C_router for i in range(N)).
Report per-task metrics and average speed-up across tasks.

Common pitfalls

• Speed-up values are projected based on router profiling and simulated 2PC environments, not real-world distributed MPC deployments.
• Accuracy drops on syntactically strict tasks (e.g., CoLA) when smaller experts are selected, highlighting a capacity-accuracy trade-off.
• MPC latency is highly sensitive to network bandwidth (10 Gbps simulated) and hardware configuration, making cross-environment comparisons difficult.

Evidence (verbatim from paper)

Specifically, performance is measured using accuracy (for MNLI, RTE, SST-2, and QNLI), the F1 score (for MRPC and QQP), the Matthews correlation coefficient (for CoLA), and the average of Pearson and Spearman correlations (for STS-B).

Citation

@misc{zhang2026securerouter,
  title={SecureRouter: Encrypted Routing for Efficient Secure Inference},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.15499}
}

• arXiv: 2604.15499