Refugee Law Outcome Eval

refugee-law-outcome-eval

Empowering Refugee Claimants and their Lawyers: Using Machine Learning to Examine Decision-Making in Refugee Law — Barale (2023) (arXiv:2308.11531, 2023)

What this evaluates

Evaluates NLP models' ability to perform legal information extraction (NER) and predict refugee claim decision outcomes from Canadian legal documents. It probes the model's capacity to handle domain-specific terminology, extract structured entities from unstructured text, and classify case outcomes based on judicial reasoning.

Datasets

• Canadian Refugee Status Determination (RSD) Cases — total 59112; splits: gold-annotated (2436), silver-annotated (25000)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted case outcomes (granted, denied, or uncertain) out of the total number of cases evaluated after applying a majority vote mechanism across sentence-level predictions.

Input / output format

Input: Semi-structured case cover metadata and full-text body of Canadian refugee status determination decisions.

Output: NER labels (DATE, ORG, GPE, PERSON, NORP, LAW, CLAIMANT_INFO, CLAIMANT_EVENT, PROCEDURE, DOC_EVIDENCE, EXPLANATION, DETERMINATION, CREDIBILITY) and a ternary decision outcome label (granted/1, denied/0, uncertain/2).

Scoring recipe

def compute_accuracy(predictions, gold):
    # predictions and gold are lists of case-level outcomes
    correct = sum(p == g for p, g in zip(predictions, gold))
    return correct / len(gold)

# Pipeline steps implied by paper:
# 1. Extract 'Determination' sentences via NER
# 2. Run BERT classifier on sentences
# 3. Apply majority vote per case
# 4. Filter cases where sentence weights fall between 0.4-0.6 as 'uncertain'
# 5. Compare final case labels against gold outcomes

Common pitfalls

• The dataset lacks explicit train/validation/test splits, making it difficult to reproduce the reported 90% accuracy without the original data partitioning or code.
• Silver-standard outcome labels are generated via a multi-step pipeline (NER extraction -> BERT classification -> majority vote) rather than direct human annotation, which may propagate NER errors into the outcome labels.

Evidence (verbatim from paper)

We then trained a classifier on 2,360 labeled sentences with positive or negative outcomes... using all of the extracted sentences. Since there may be multiple extracted sentences per case, we employed a majority vote mechanism to determine the outcome of each case. Sentences that could not be confidently classified as positive or negative (with a computed weight between 0.4 and 0.6) were categorized as 'Uncertain'. Our classifier achieved 90% accuracy, classifying 52,234 sentences into three categories: granted (1), denied (0), and uncertain (2)

Citation

@misc{barale2023empowering,
  title={Empowering Refugee Claimants and their Lawyers: Using Machine Learning to Examine Decision-Making in Refugee Law},
  author={Barale (2023)},
  year={2023},
  note={arXiv:2308.11531}
}

• arXiv: 2308.11531