Workrb Eval

workrb-eval

WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain — De Lange et al. (2026) (arXiv:2604.13055, 2026)

What this evaluates

Evaluates AI models on work-domain recommendation and NLP tasks, primarily focusing on ranking and retrieval scenarios such as occupation-to-skill matching, candidate recommendation, and skill/job normalization. It tests cross-lingual and multilingual retrieval capabilities over standardized occupational ontologies like ESCO.

Datasets

• ESCO Occupation-to-Skill — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• ESCO Skill-to-Occupation — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• Job Title Sim. — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• SkillMatch-1K — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• Query-Candidate — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• Project-Candidate — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• JobBERT — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• MELO — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• ESCO Alternatives — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• MELS — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• House — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• Tech — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB
• SkillSkape — total ?; splits: test (-1); repo https://github.com/techwolf-ai/WorkRB

Metrics

• MAP (primary) — range: percent
  • Mean Average Precision across ranked lists of targets for each query. Calculated as the mean of the average precision scores for all queries, where average precision is the precision at each relevant position averaged over the ranked list.

Input / output format

Input: Query (e.g., occupation title, skill name, job description, or candidate profile) and a target space (e.g., list of ESCO skills, occupations, or candidate profiles) to be ranked.

Output: Ranked list of target items ordered by relevance to the query.

Scoring recipe

def compute_map(predictions, gold):
    hits = 0
    ap = 0.0
    for i, pred in enumerate(predictions):
        if pred in gold:
            hits += 1
            ap += hits / (i + 1)
    return ap / len(gold) if len(gold) > 0 else 0.0
# Final metric is mean of ap across all queries

Common pitfalls

• Tasks are formulated as ranking problems, not classification, so models must output ordered lists rather than binary labels.
• Cross-lingual setups require independent resolution of query and target spaces per language, which can lead to mismatched ontologies if not handled dynamically.
• Evaluation supports both monolingual and cross-lingual configurations; aggregating metrics across languages requires explicit strategy definition.

Evidence (verbatim from paper)

Following prior work, all of the contributed work-domain tasks are formulated as ranking problems. Table 2 reports mean average precision (MAP) scores across all task groups for a representative subset of baselines.

Citation

@misc{delange2026workrb,
  title={WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain},
  author={De Lange et al. (2026)},
  year={2026},
  note={arXiv:2604.13055}
}

• arXiv: 2604.13055