Ne Classification Bootstrapping Eval

ne-classification-bootstrapping-eval

Lightly-supervised Representation Learning with Global Interpretability — Valenzuela-Escárcega et al. (2018) (arXiv:1805.11545, 2018)

What this evaluates

Evaluates lightly-supervised representation learning and pattern-based extraction for named entity classification. It probes the model's ability to learn custom entity and pattern embeddings via bootstrapping, and to derive an interpretable global decision list for classification without using gold labels during training.

Datasets

• CoNLL-2003 — total ?; splits: train (-1), dev (-1)
• Ontonotes — total ?; splits: train (-1), dev (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Standard entity-level F1-score for named entity classification. (Exact metric not explicitly specified in the text, but F1 is the standard evaluation metric for this task.)

Input / output format

Input: Free text documents containing marked entity boundaries. The system is initialized with 10 manually selected seed entities per category.

Output: Custom entity and pattern embeddings, and a ranked global decision list of n-gram patterns (up to 4 tokens) used to classify entities into categories.

Scoring recipe

def compute_f1(gold_entities, pred_entities):
    tp = len(gold_entities & pred_entities)
    fp = len(pred_entities - gold_entities)
    fn = len(gold_entities - pred_entities)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Gold labels are explicitly excluded during training; only entity boundaries are used.
• Hyperparameters are tuned on the development set, but final experiments are run on the training partition, deviating from standard train/val/test protocols.
• Seed selection is manual based on frequency, which can introduce bias and affect bootstrapping convergence.

Evidence (verbatim from paper)

We evaluate the above algorithms on the task of named entity classification from free text. ... These datasets contain marked entity boundaries with labels for each marked entity. Here we only use the entity boundaries but not the labels of these entities during the training of our bootstrapping systems. To simulate learning from large texts, we tuned hyper parameters on development, but ran the actual experiments on the train partitions.

Citation

@misc{valenzuelaescarcega2018lightly,
  title={Lightly-supervised Representation Learning with Global Interpretability},
  author={Valenzuela-Escárcega et al. (2018)},
  year={2018},
  note={arXiv:1805.11545}
}

• arXiv: 1805.11545