Coda Eval

coda-eval

CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP — Evuru et al. (2024) (arXiv:2404.00415, 2024)

What this evaluates

Evaluates a training-free, constraint-based data augmentation framework for low-resource NLP. It probes whether synthetically augmented data improves downstream performance across sequence classification, intent classification, named entity recognition, and question answering tasks compared to gold-only and other augmentation baselines.

Datasets

• Huffpost — total ?; splits: train (-1)
• Yahoo — total ?; splits: train (-1)
• OTS — total ?; splits: train (-1)
• ATIS — total ?; splits: train (-1)
• Massive — total ?; splits: train (-1)
• ConLL-2003 — total ?; splits: train (-1)
• OntoNotes-5.0 — total ?; splits: train (-1)
• EBMNLP — total ?; splits: train (-1)
• BC2GM — total ?; splits: train (-1)
• SQuAD — total ?; splits: train (-1)
• NewsQA — total ?; splits: train (-1)

Metrics

• micro-average F1 score (primary) — range: [0, 1]
  • Aggregates true positives, false positives, and false negatives across all classes before computing precision and recall. F1 is calculated as 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Text instances for downstream NLU tasks (sequence classification, intent classification, NER, QA) with low-resource training splits (e.g., 50, 100, 200, 500, or 1000 examples).

Output: Predicted class labels, entity tags, or answers corresponding to the input instances.

Scoring recipe

tp, fp, fn = 0, 0, 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold: tp += 1
    else: fp += 1; fn += 1
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
micro_f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return micro_f1  # averaged over 3 runs × 3 seeds

Common pitfalls

• Reporting macro-F1 instead of the specified micro-average F1.
• Failing to average results across the 3 runs and 3 random seeds as mandated.
• Using full-dataset splits instead of the specified low-resource training sizes (50, 100, 200, 500, 1000).

Evidence (verbatim from paper)

We report the micro-average F1 score averaged across 3 runs for 3 random seeds.

Citation

@misc{evuru2024coda,
  title={CoDa: Constrained Generation based Data Augmentation for Low-Resource NLP},
  author={Evuru et al. (2024)},
  year={2024},
  note={arXiv:2404.00415}
}

• arXiv: 2404.00415