Dyad Arch Eval

dyad-arch-eval

DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layers — Chandy et al. (2023) (arXiv:2312.06881, 2023)

What this evaluates

Evaluates a block-sparse linear layer approximation (DYAD) against dense baselines across standard NLP and vision benchmarks, measuring accuracy preservation and computational efficiency.

Datasets

• BLIMP — total ?; splits: test (-1)
• OPENLLM — total ?; splits: test (-1)
• GLUE+ — total ?; splits: test (-1)
• MNIST — total ?; splits: test (-1)

Metrics

• BLIMP accuracy (primary) — range: percent
  • Percentage of minimal pairs where the model assigns a higher probability to the grammatical sentence over the ungrammatical one.
• GLUE+ score — range: percent
  • Macro-averaged score across 7 GLUE tasks plus WSC and BOOLQ, evaluated post-finetuning.
• OPENLLM accuracy — range: percent
  • Average accuracy across 4 few-shot openbook multiple-choice benchmarks, computed via lm-eval-harness.
• Training time per minibatch — range: other
  • Wall-clock time in milliseconds for forward and backward passes per minibatch, reported for total model and FF-only modules.
• In-Training GPU Memory Usage — range: other
  • Peak dynamic GPU memory consumption during training, including optimizer states and cached activations.

Input / output format

Input: Sentence pairs (BLIMP), few-shot multiple-choice prompts (OPENLLM), NLU task prompts (GLUE+), and MNIST digit images.

Output: Probability or log-probability for grammatical preference, selected answer choice, predicted class label, or profiling counters for time and memory.

Scoring recipe

def evaluate(predictions, golds, timings, memory_samples):
    # Accuracy for BLIMP, OPENLLM, GLUE+
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    acc = correct / len(golds)
    # GLUE+ aggregation (macro-average of task-specific metrics)
    glue_score = np.mean([task_metric(p, g) for p, g in glue_tasks])
    # Efficiency metrics
    avg_time_ms = np.mean(timings)
    peak_mem_gb = max(memory_samples)
    return {'accuracy': acc, 'glue_score': glue_score, 'time_ms': avg_time_ms, 'mem_gb': peak_mem_gb}

Common pitfalls

• Confusing the pretraining corpus (BABYLM) with the actual evaluation benchmarks used for reporting results.
• Overlooking the sequential processing overhead in the base DYAD-IT forward pass, which can negate theoretical speedups unless the -CAT parallel optimization is applied.
• Comparing DYAD variants directly to DENSE-EXT without accounting for differences in pretraining scale (10M vs 100M tokens) or initialization.

Evidence (verbatim from paper)

Zero-Shot: BLIMP Benchmark of Linguistic Minimal Pairs (BLIMP) [16] consists of pairs of grammatical-ungrammatical sentences grouped by 12 broad phenomena e.g. anaphora and noun-verb agreement. A good LLM ought to assign higher probability to the grammatical member. Few-Shot: OPENLLM The OPENLLM leaderboard [17] has become a prevalent way to benchmark LLMs based on 4 few-shot openbook MCQesque benchmarks. Internally, it uses LMEvalHarness [18], which we replicate to compute numbers for our models as well as BabyLm's pretrained checkpoints. Finetuned: GLUE+ General Lang. Understanding Eval (GLUE) [19], is a set of 7 NLU tasks, each evaluated post-finetuning.

Citation

@misc{chandy2023dyad,
  title={DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layers},
  author={Chandy et al. (2023)},
  year={2023},
  note={arXiv:2312.06881}
}

• arXiv: 2312.06881