Hulk Eval

hulk-eval

HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing — Zhou et al. (2020) (arXiv:2002.05829, 2020)

What this evaluates

Evaluates the computational efficiency and energy cost of NLP models across pretraining, fine-tuning, and inference phases. It measures the time and monetary cost required to reach predefined performance thresholds on standard NLP tasks, normalized against a BERT-Large baseline.

Datasets

• CoNLL 2003 — total 17291; splits: train (14041), dev (3250)
• MNLI — total 412349; splits: train (392702), dev (19647)
• SST-2 — total 68221; splits: train (67349), dev (872)

Metrics

• efficiency score (primary) — range: other
  • Normalized efficiency score computed as the ratio of BERT-Large's time/cost to the model's time/cost for each task. The overall score is the sum of these normalized ratios across CoNLL-2003, SST-2, and MNLI. Models must first meet a task-specific F1 or Accuracy cut-off to be scored.

Input / output format

Input: Raw text sequences (sentences or tokenized documents) for named entity recognition, natural language inference, or sentiment classification.

Output: Token-level entity labels for CoNLL-2003; class labels (entailment/contradiction/neutral or positive/negative) for MNLI and SST-2.

Scoring recipe

def compute_efficiency_score(predictions, gold, time_model, time_bert_large):
    # Check if model meets task-specific cut-off
    if task == 'CoNLL-2003':
        if f1_score(gold, predictions) < 91: return 0.0
    elif task in ['MNLI', 'SST-2']:
        cutoff = 90 if task == 'SST-2' else 85
        if accuracy_score(gold, predictions) < cutoff: return 0.0
    # Compute normalized efficiency ratio
    return time_bert_large / time_model

Common pitfalls

• Models that fail to reach the performance cut-off within the allowed epochs are marked N/A or scored 0, heavily penalizing them regardless of their actual inference speed.
• The efficiency score is strictly relative to BERT-Large; a score >1 means faster than BERT-Large, but absolute efficiency cannot be compared across different hardware setups without normalization.
• Time measurements include only the specific phase (pretraining/fine-tuning/inference) and exclude data loading or framework overhead, which can skew real-world cost estimates.

Evidence (verbatim from paper)

the efficiency score on different tasks is defined as the sum of normalized time and cost. Here we normalize the time and cost because they vary dramatically between tasks. In order to simplify the process, we compute the ratio of BERT_LARGE's time and cost to that of each model as the normalized measure as shown in Table 3 and Table 4.

Citation

@misc{zhou2020hulk,
  title={HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing},
  author={Zhou et al. (2020)},
  year={2020},
  note={arXiv:2002.05829}
}

• arXiv: 2002.05829