# Dyad Arch Eval

> Evaluates a block-sparse linear layer approximation (DYAD) against dense baselines across standard NLP and vision benchmarks, measuring accuracy preservation and computational efficiency. Use when the user wants to benchmark on BLIMP, OPENLLM, GLUE+, MNIST, or asks about evaluating this task. Reports BLIMP accuracy.

- Skill: `qhjqhj00/dyad-arch-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/dyad-arch-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/dyad-arch-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/dyad-arch-eval

---


# dyad-arch-eval

> DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layers — Chandy et al. (2023) (arXiv:2312.06881, 2023)

## What this evaluates

Evaluates a block-sparse linear layer approximation (DYAD) against dense baselines across standard NLP and vision benchmarks, measuring accuracy preservation and computational efficiency.

## Datasets

- **BLIMP** — total ?; splits: test (-1)
- **OPENLLM** — total ?; splits: test (-1)
- **GLUE+** — total ?; splits: test (-1)
- **MNIST** — total ?; splits: test (-1)

## Metrics

- `BLIMP accuracy` **(primary)** — range: percent
  - Percentage of minimal pairs where the model assigns a higher probability to the grammatical sentence over the ungrammatical one.
- `GLUE+ score` — range: percent
  - Macro-averaged score across 7 GLUE tasks plus WSC and BOOLQ, evaluated post-finetuning.
- `OPENLLM accuracy` — range: percent
  - Average accuracy across 4 few-shot openbook multiple-choice benchmarks, computed via lm-eval-harness.
- `Training time per minibatch` — range: other
  - Wall-clock time in milliseconds for forward and backward passes per minibatch, reported for total model and FF-only modules.
- `In-Training GPU Memory Usage` — range: other
  - Peak dynamic GPU memory consumption during training, including optimizer states and cached activations.

## Input / output format

**Input**: Sentence pairs (BLIMP), few-shot multiple-choice prompts (OPENLLM), NLU task prompts (GLUE+), and MNIST digit images.

**Output**: Probability or log-probability for grammatical preference, selected answer choice, predicted class label, or profiling counters for time and memory.

## Scoring recipe

```python
def evaluate(predictions, golds, timings, memory_samples):
    # Accuracy for BLIMP, OPENLLM, GLUE+
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    acc = correct / len(golds)
    # GLUE+ aggregation (macro-average of task-specific metrics)
    glue_score = np.mean([task_metric(p, g) for p, g in glue_tasks])
    # Efficiency metrics
    avg_time_ms = np.mean(timings)
    peak_mem_gb = max(memory_samples)
    return {'accuracy': acc, 'glue_score': glue_score, 'time_ms': avg_time_ms, 'mem_gb': peak_mem_gb}
```

## Common pitfalls

- Confusing the pretraining corpus (BABYLM) with the actual evaluation benchmarks used for reporting results.
- Overlooking the sequential processing overhead in the base DYAD-IT forward pass, which can negate theoretical speedups unless the -CAT parallel optimization is applied.
- Comparing DYAD variants directly to DENSE-EXT without accounting for differences in pretraining scale (10M vs 100M tokens) or initialization.

## Evidence (verbatim from paper)

> Zero-Shot: BLIMP Benchmark of Linguistic Minimal Pairs (BLIMP) [16] consists of pairs of grammatical-ungrammatical sentences grouped by 12 broad phenomena e.g. anaphora and noun-verb agreement. A good LLM ought to assign higher probability to the grammatical member. Few-Shot: OPENLLM The OPENLLM leaderboard [17] has become a prevalent way to benchmark LLMs based on 4 few-shot openbook MCQesque benchmarks. Internally, it uses LMEvalHarness [18], which we replicate to compute numbers for our models as well as BabyLm's pretrained checkpoints. Finetuned: GLUE+ General Lang. Understanding Eval (GLUE) [19], is a set of 7 NLU tasks, each evaluated post-finetuning.

## Citation

```bibtex
@misc{chandy2023dyad,
  title={DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layers},
  author={Chandy et al. (2023)},
  year={2023},
  note={arXiv:2312.06881}
}
```

- arXiv: 2312.06881