Tanda Augmentation Eval

tanda-augmentation-eval

Learning to Compose Domain-Specific Transformations for Data Augmentation — Ratner et al. (2017) (arXiv:1709.01643, 2017)

What this evaluates

Evaluates whether automatically composing domain-specific data augmentation transformations improves end-task classification performance. It probes the ability of a learned sequence model to generate effective augmentation pipelines compared to heuristic or random baselines across image and text domains.

Datasets

• MNIST — total ?; splits: test (-1); HF mnist
• CIFAR-10 — total ?; splits: test (-1); HF cifar10
• ACE (Employment relation extraction) — total ?; splits: test (-1)
• DDSM (Mammography) — total 1506; splits: test (-1)

Metrics

• test set accuracy (primary) — range: percent
  • Percentage of correctly classified instances in the test set. Computed as (number of correct predictions) / (total test instances).
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for the positive class, used specifically for the ACE relation extraction task.

Input / output format

Input: Labeled training data (subsampled), unlabeled data for generator training, and held-out test set. The end classifier receives augmented training instances generated by composing transformation functions.

Output: Class labels predicted by the end classifier for each test instance.

Scoring recipe

def compute_metric(predictions, gold, task):
    if task == 'ACE':
        tp = sum(1 for p, g in zip(predictions, gold) if p == g == 'positive')
        fp = sum(1 for p, g in zip(predictions, gold) if p == 'positive' and g != 'positive')
        fn = sum(1 for p, g in zip(predictions, gold) if p != 'positive' and g == 'positive')
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

Common pitfalls

• Using the full labeled dataset for augmentation training instead of the specified unlabeled subset.
• Failing to apply the task-specific sequence lengths (L=5 for ACE, L=7 for DDSM+DS, L=10 for others) calibrated during initial experiments.
• Comparing against baselines without including the same random cropping/Basic augmentation step used by the proposed method.

Evidence (verbatim from paper)

We present these results in Table 2, where we report test set accuracy (or F1 score for ACE), and use a random subsample of the available labeled training data.

Citation

@misc{ratner2017tanda,
  title={Learning to Compose Domain-Specific Transformations for Data Augmentation},
  author={Ratner et al. (2017)},
  year={2017},
  note={arXiv:1709.01643}
}

• arXiv: 1709.01643