Lexglue Eval

lexglue-eval

An energy-based comparative analysis of common approaches to text classification in the Legal domain — Gultekin et al. (2023) (arXiv:2311.01256, 2023)

What this evaluates

Evaluates text classification performance and environmental impact (energy, cost, emissions) of NLP models on legal domain datasets. It compares traditional machine learning approaches against transformer-based models across multiple legal benchmarks.

Datasets

• LexGLUE — total ?; splits: test (-1); repo https://github.com/coastalcph/lex-glue

Metrics

• mF1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, averaged across all instances.
• MF1 — range: [0, 1]
  • Harmonic mean of precision and recall, averaged across all classes.
• KWh — range: other
  • Total electrical energy consumed during model training or inference, measured via hardware monitoring tools.
• € — range: other
  • Monetary cost of electricity consumption, calculated using local electricity rates.
• CO2 — range: other
  • Carbon dioxide equivalent emissions generated by electricity consumption, estimated using grid emission factors.

Input / output format

Input: Raw text documents from legal domain datasets (e.g., court rulings, legislation, contracts, terms of service).

Output: Predicted class label(s) for each input document.

Scoring recipe

def evaluate(predictions, gold_labels):
    micro_f1 = f1_score(gold_labels, predictions, average='micro')
    macro_f1 = f1_score(gold_labels, predictions, average='macro')
    # Energy, cost, and CO2 are measured externally via system monitors
    # and normalized relative to a baseline model (e.g., SVM_nlp)
    return micro_f1, macro_f1, energy_kwh, cost_eur, co2_kg

Common pitfalls

• Energy consumption and CO2 estimates are highly dependent on the specific hardware, runtime environment, and local electricity grid factors used during measurement.
• The paper normalizes energy/cost/CO2 relative to a baseline model (SVM_nlp or SVM_bow) rather than reporting absolute values, which complicates direct comparison with other studies.
• F1 scores are reported for both micro and macro averaging; macro F1 can be heavily influenced by class imbalance in legal datasets.

Evidence (verbatim from paper)

In the following, we report the comparative analysis individually for each dataset in terms of (a) performance, using the F1 score, both micro (mF1) and macro (MF1) averaging, and (b) energy consumption (KWh), costs (€) and carbon footprint (CO2) estimated for each experiment.

Citation

@misc{gultekin2023energy,
  title={An energy-based comparative analysis of common approaches to text classification in the Legal domain},
  author={Gultekin et al. (2023)},
  year={2023},
  note={arXiv:2311.01256}
}

• arXiv: 2311.01256