Nash Pruning Eval

nash-pruning-eval

NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models — Ko et al. (2023) (arXiv:2310.10054, 2023)

What this evaluates

Evaluates the generation quality and inference efficiency of structuredly pruned encoder-decoder language models across abstractive QA, summarization, classification, and instruction-following tasks.

Datasets

• TweetQA — total ?; splits: (unstated)
• XSum — total ?; splits: (unstated)
• SAMSum — total ?; splits: (unstated)
• CNN/DailyMail — total ?; splits: (unstated)
• GLUE/SuperGLUE (RTE, BoolQ, CB) — total ?; splits: (unstated)
• Databricks-dolly-15k — total 15000; splits: train (14000), eval (1000)
• Self-Instruct — total ?; splits: (unstated)
• Vicuna Evaluation — total ?; splits: (unstated)

Metrics

• METEOR — range: [0, 1]
  • Standard METEOR score measuring alignment between generated and reference text using synonyms, stems, and exact matches. Used for abstractive QA.
• ROUGE-L (primary) — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation based on the longest common subsequence. Used for summarization and instruction-following tasks.
• Accuracy — range: [0, 1]
  • Exact match ratio between predicted and gold class labels for classification tasks (RTE, BoolQ, CB).
• Speedup — range: other
  • Ratio of baseline model inference latency to pruned model inference latency, measured in wall-clock time on GPU.

Input / output format

Input: Source text, question, or instruction prompt passed to the encoder.

Output: Generated target text or classification label produced by the decoder.

Scoring recipe

def compute_metrics(predictions, golds, base_latency, pruned_latency):
    meteor = sum(meteor_score(g, p) for g, p in zip(golds, predictions)) / len(golds)
    rouge_l = sum(rouge_l_score(g, p) for g, p in zip(golds, predictions)) / len(golds)
    acc = sum(1.0 if g == p else 0.0 for g, p in zip(golds, predictions)) / len(golds)
    speedup = base_latency / pruned_latency
    return {'METEOR': meteor, 'ROUGE-L': rouge_l, 'Accuracy': acc, 'Speedup': speedup}

Common pitfalls

• The paper reports results on validation sets rather than held-out test sets for all datasets.
• Sparsity is calculated excluding embedding parameters, which differs from standard parameter-counting conventions.
• Speedup is measured as wall-clock latency ratio against the unpruned T5-Base baseline, not theoretical FLOPs reduction.

Evidence (verbatim from paper)

We evaluate the output quality using METEOR (Banerjee and Lavie, 2005) for abstractive question answering and ROUGE (Lin, 2004) for the summarization tasks. The reported results are based on the validation sets of all datasets.

Citation

@misc{ko2023nash,
  title={NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models},
  author={Ko et al. (2023)},
  year={2023},
  note={arXiv:2310.10054}
}

• arXiv: 2310.10054