Jobbert Eval

jobbert-eval

JobBERT: Understanding Job Titles through Skills — Decorte et al. (2021) (arXiv:2109.09605, 2021)

What this evaluates

Evaluates a model's ability to normalize job titles by mapping them to standardized ESCO occupation labels using semantic similarity. It probes the model's capacity to handle hierarchical occupational taxonomies and filter out irrelevant contextual tokens like locations.

Datasets

• JobBERT Vacancy Titles — total 30926; splits: validation (15463), test (15463); repo https://github.com/jensjorisdecorte/JobBERT-evaluation-dataset

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: For each job title query, rank the candidate ESCO occupations by predicted similarity. The score for a query is 1 divided by the rank of the first correct ESCO label. MRR is the average of these scores across all queries.

Input / output format

Input: A single English vacancy title string.

Output: A ranked list of candidate ESCO occupation labels, or a single predicted ESCO label.

Scoring recipe

def compute_mrr(predictions, gold_labels):
    scores = []
    for pred, gold in zip(predictions, gold_labels):
        rank = next(i for i, c in enumerate(pred) if c == gold) + 1
        scores.append(1.0 / rank)
    return sum(scores) / len(scores)

Common pitfalls

• The dataset only contains unique titles, so duplicate variations are not represented in the evaluation split.
• MRR assumes a single correct ESCO label per title, but the paper notes an upper bound of 65% ambiguity, meaning some titles may legitimately map to multiple occupations.
• The evaluation uses a stratified split based on ESCO label distribution, which may not reflect real-world frequency distributions of job titles.

Evidence (verbatim from paper)

The final dataset contains a total of 30,926 unique English vacancy titles, each tagged with one of the remaining ESCO occupations. We make a stratified split of this dataset based on the ESCO label distribution into two equally sized parts, for validation and testing. Throughout the training, the performance is measured on the validation set in terms of mean reciprocal rank (MRR) after each multiple of 5k batches.

Citation

@misc{decorte2021jobbert,
  title={JobBERT: Understanding Job Titles through Skills},
  author={Decorte et al. (2021)},
  year={2021},
  note={arXiv:2109.09605}
}

• arXiv: 2109.09605