Core Fewshot Rc Eval

core-fewshot-rc-eval

CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation — Borchert et al. (2023) (arXiv:2310.12024, 2023)

What this evaluates

Evaluates few-shot relation classification models on company and business entity relations, testing their ability to resolve entity ambiguity and adapt across domains using limited labeled examples.

Datasets

• CORE — total ?; splits: train (-1), test (-1); repo https://github.com/pnborchert/CORE

Metrics

• Micro F1 (primary) — range: [0, 1]
  • Standard micro-averaged F1 score: aggregates true positives, false positives, and false negatives across all classes before computing precision and recall.
• Macro F1 — range: [0, 1]
  • Standard macro-averaged F1 score: computes the F1 score for each class independently and then takes the unweighted mean.

Input / output format

Input: A text sentence containing two named entities, with a relation label from a set of N+1 relations (including NOTA).

Output: Predicted relation label for the entity pair.

Scoring recipe

def compute_f1(y_true, y_pred, average='micro'):
    from sklearn.metrics import f1_score
    return f1_score(y_true, y_pred, average=average, zero_division=0)

Common pitfalls

• NOTA (None Of The Above) relation handling varies across datasets; this paper enforces a 50% NOTA rate for FewRel to match prior work.
• Episode sampling uses disjoint relation sets for train and test, meaning models never see test relations during training, testing true few-shot generalization.
• Micro vs Macro F1 can diverge significantly in imbalanced or NOTA-heavy settings; the paper reports both to capture class-level vs global performance.

Evidence (verbatim from paper)

We adopt the N-way K-shot evaluation setting, where the model is exposed to a limited number of instances $N imes K$ during training. ... We evaluate RC models in the few-shot domain adaptation setting for CORE, FewRel and TA-CRED. ... Table 3: Micro F1 / Macro F1 for in-domain and out-of-domain models evaluated on CORE.

Citation

@misc{borchert2023core,
  title={CORE: A Few-Shot Company Relation Classification Dataset for Robust Domain Adaptation},
  author={Borchert et al. (2023)},
  year={2023},
  note={arXiv:2310.12024}
}

• arXiv: 2310.12024