Swiss Judgment Prediction Eval

swiss-judgment-prediction-eval

An Empirical Study on Cross-X Transfer for Legal Judgment Prediction — Niklaus et al. (2022) (arXiv:2209.12325, 2022)

What this evaluates

Evaluates Legal Judgment Prediction (LJP) models across languages (German, French, Italian), regions, and legal domains. It probes cross-lingual, cross-regional, and cross-domain transfer capabilities, as well as the impact of data augmentation and adapter-based fine-tuning on model fairness and accuracy.

Datasets

• SJP (Swiss Judgment Prediction) — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/JoelNiklaus/Swiss

Metrics

• macro-averaged F1 score (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across the approval and dismissal classes to account for high class-imbalance (~20/80 ratio).

Input / output format

Input: Legal case documents (up to 2048 tokens), processed as consecutive 512-token blocks for Hierarchical BERT encoding.

Output: Binary classification label: approval or dismissal.

Scoring recipe

def compute_macro_f1(gold, preds):
    # Calculate precision and recall for each class
    # Average them across classes, then compute F1
    return f1_score(gold, preds, average='macro')

Common pitfalls

• The dataset has high class imbalance (~20/80 approval/dismissal ratio), so accuracy is misleading; macro-F1 is strictly required.
• Documents exceed 512 tokens (90% are up to 2048), requiring hierarchical block encoding (4x512) instead of standard BERT truncation.
• Results are averaged across 3 random seeds; single-run scores are not comparable to reported values.

Evidence (verbatim from paper)

We follow Niklaus et al. (2021) and report macro-averaged F1 score to account for the high class-imbalance in the dataset (approx. 20/80 approval/dismissal ratio). We repeat each experiment with 3 different random seeds and report the average score and standard deviation across runs (seeds).

Citation

@misc{niklaus2022empirical,
  title={An Empirical Study on Cross-X Transfer for Legal Judgment Prediction},
  author={Niklaus et al. (2022)},
  year={2022},
  note={arXiv:2209.12325}
}

• arXiv: 2209.12325