Fewjoint Eval

fewjoint-eval

FewJoint: A Few-shot Learning Benchmark for Joint Language Understanding — Hou et al. (2020) (arXiv:2009.08138, 2020)

What this evaluates

Evaluates few-shot joint language understanding by measuring a model's ability to simultaneously predict dialogue intent and extract slots from query sentences using only a small support set from unseen domains.

Datasets

• FewJoint — total ?; splits: test (-1), dev (-1); repo https://github.com/AtmaHou/MetaDialog

Metrics

• Intent Accuracy — range: [0, 1]
  • Percentage of query sentences correctly classified into the true intent label.
• Slot F1-score — range: [0, 1]
  • F1-score computed on slot tags using the conlleval script, calculated only on query samples.
• Sentence Accuracy (primary) — range: [0, 1]
  • Proportion of query sentences where both the predicted intent and all predicted slot tags exactly match the ground truth.

Input / output format

Input: A support set containing few labeled examples (intent + slots) from a target domain, and a query sentence to be classified/tagged.

Output: Predicted intent label and BIO slot tags for the query sentence.

Scoring recipe

def score(predictions, gold):
    intent_correct = sum(1 for p, g in zip(predictions, gold) if p.intent == g.intent)
    intent_acc = intent_correct / len(gold)
    slot_f1 = conlleval_f1([p.slots for p in predictions], [g.slots for g in gold])
    sent_correct = sum(1 for p, g in zip(predictions, gold) if p.intent == g.intent and p.slots == g.slots)
    sent_acc = sent_correct / len(gold)
    return {'Intent Accuracy': intent_acc, 'Slot F1': slot_f1, 'Sentence Accuracy': sent_acc}

Common pitfalls

• Evaluating on the support set instead of the held-out query set.
• Computing slot F1 without the strict BIO formatting rules enforced by the conlleval script.
• Reporting results from a single random seed instead of averaging over 5 seeds as specified.

Evidence (verbatim from paper)

There are three main metrics for evaluation: Intent Accuracy, Slot F1-score, Sentence Accuracy. Specifically, we calculate the Slot F1-score on query samples with conlleval script. For Sentence Accuracy, we consider that one sentence is correct only when all its slots and intent are correct, and vice versa. All models are evaluated on the same support-query pairs for fairness.

Citation

@misc{hou2020fewjoint,
  title={FewJoint: A Few-shot Learning Benchmark for Joint Language Understanding},
  author={Hou et al. (2020)},
  year={2020},
  note={arXiv:2009.08138}
}

• arXiv: 2009.08138