Jobfair Behavioral Transfer Eval

jobfair-behavioral-transfer-eval

Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context — Sorstkins et al. (2025) (arXiv:2509.15366, 2025)

What this evaluates

Evaluates behavioral cloning and extraction fidelity in bias-mitigation LLM agents processing job descriptions. It measures how well mutated agent outputs align with expert golden references using cross-item similarity and a multi-facet diagnostic rubric.

Datasets

• JobFair Corpus — total 300; splits: test (300)

Metrics

• BERTScore (F1) against 5-NN average (primary) — range: [0, 1]
  • BERTScore-F1 (using roberta-base) computed between system output and each of the top-5 nearest golden exemplars per item, then averaged. The reported outcome is the delta: mutated_avg minus original_avg.
• EDScore — range: [0, 1]
  • Weighted sum of 6 facet scores (Correctness 0.35, Completeness 0.30, Over-Extraction 0.15, Detail Accuracy 0.10, Terminology Consistency 0.05, Reasoning Alignment 0.05). Each facet is scored 0-1.
• BDScore — range: [0, 5]
  • Equal-weighted sum of 4 facet scores (tone match, stylistic fidelity, manner of expression, semantic alignment), each scored 0-5.

Input / output format

Input: English-language job descriptions from a curated corpus. For behavioral transfer: each item yields two agent-generated tracks ('Expert Suggestion' and 'Comment Suggestion'). For extraction: 13 documents sampled to cover job profile domains and length deciles.

Output: Agent-generated text suggestions/comments or extracted entities/facets. Evaluated against golden exemplars or expert annotations.

Scoring recipe

def score_bertscore_5nn(output, golden_pool):
    top5 = get_top_5_nearest_golden(output, golden_pool)
    scores = [bertscore_f1(output, g, encoder='roberta-base') for g in top5]
    return sum(scores) / len(scores)

def score_ed(facet_scores):
    weights = {'Correctness': 0.35, 'Completeness': 0.30, 'Over-Extraction': 0.15,
               'Detail Accuracy': 0.10, 'Terminology Consistency': 0.05, 'Reasoning Alignment': 0.05}
    return sum(facet_scores[facet] * w for facet, w in weights.items())

Common pitfalls

• BERTScore ceiling effect: 5-NN averaging and cross-item matching compress the score range, preventing scores from reaching 1.0 and masking qualitative improvements.
• LLM-as-a-Judge arithmetic failure: The judge reliably scores individual facets but consistently fails at deterministic weighted aggregation; final scores must be computed in code, not by the LLM.
• Golden-set heterogeneity: 'Comment' references are longer and more actionable than 'Expert' cues, creating a stronger learning signal and inflating effect sizes for the Comment track.

Evidence (verbatim from paper)

Similarity is measured with BERTScore-F1 using roberta-base as the encoder. For each item, the score is computed against the 5-NN average—the mean BERTScore to the five most similar golden exemplars drawn from other jobs—yielding a cross-item, multi-reference evaluation.

Citation

@misc{sorstkins2025diagnostics,
  title={Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context},
  author={Sorstkins et al. (2025)},
  year={2025},
  note={arXiv:2509.15366}
}

• arXiv: 2509.15366