Routing Algorithm Benchmarks Eval

routing-algorithm-benchmarks-eval

An Algorithm for Routing Vectors in Sequences — Heinsen (2022) (arXiv:2211.11754, 2022)

What this evaluates

Evaluates the routing algorithm's efficiency, scalability, and classification accuracy on standard NLP and vision benchmarks when used as a classification head over frozen pretrained Transformers.

Datasets

• IMDB — total ?; splits: test (-1)
• SST-5 — total ?; splits: test (-1)
• SST-2 — total ?; splits: test (-1)
• ImageNet-1K — total ?; splits: test (-1)
• CIFAR-100 — total ?; splits: test (-1)
• CIFAR-10 — total ?; splits: test (-1)

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly classified instances out of the total number of instances.
• Parameter count — range: other
  • Total number of trainable parameters in the routing implementation.
• Memory footprint — range: other
  • Peak memory allocation during a forward pass with gradient tracking.
• Execution time — range: other
  • Wall-clock time for a forward pass on a GPU at 32-bit precision.

Input / output format

Input: Flattened sequence of hidden embeddings from a frozen pretrained Transformer (RoBERTa-large for NLP, BEiT-large for vision), processed through three sequential routing layers.

Output: Classification scores for each class (1D output vectors), plus end-to-end credit assignment matrices.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_efficiency(model, seq_len, vec_dim):
    params = count_parameters(model)
    memory = peak_memory_allocation(model, seq_len, vec_dim)
    time = measure_forward_pass_time(model, seq_len, vec_dim)
    return params, memory, time

Common pitfalls

• Efficiency comparisons restrict the routing algorithm to equal input/output sequence lengths and exactly 2 iterations to match a single Transformer encoder layer, which may underrepresent its typical usage.
• Credit assignments are computed by multiplying three routing matrices and scaling by the standard deviation of the product, rather than using standard attention or gradient-based attribution.
• The classification head flattens hidden states from all Transformer layers, creating variable-length sequences that require chunking if exceeding the Transformer's max length.

Evidence (verbatim from paper)

We test our implementation on six classification benchmarks in natural language and vision, obtaining accuracy that is competitive with, and in one case better than, the state of the art (Table 2).

Citation

@misc{heinsen2022routingvectors,
  title={An Algorithm for Routing Vectors in Sequences},
  author={Heinsen (2022)},
  year={2022},
  note={arXiv:2211.11754}
}

• arXiv: 2211.11754