Linguistic Probing Eval

linguistic-probing-eval

How transfer learning impacts linguistic knowledge in deep NLP models? — Durrani et al. (2021) (arXiv:2105.15179, 2021)

What this evaluates

Evaluates how fine-tuning on downstream NLP tasks redistributes linguistic knowledge across transformer layers. It probes for part-of-speech tagging, syntactic chunking, and semantic tagging capabilities using linear classifiers on layer-wise hidden states.

Datasets

• Penn TreeBank — total ?; splits: train (-1), val (-1), test (-1)
• CoNLL 2000 — total ?; splits: train (-1), val (-1), test (-1)
• Parallel Meaning Bank — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy computed on the test split for each linguistic task.
• selectivity — range: [0, 1]
  • Ratio of the probing classifier's accuracy to a random baseline accuracy, used to control for probe memorization.

Input / output format

Input: Tokenized word/subword sequences with corresponding linguistic labels (POS tags, chunk tags, or semantic tags) from the respective datasets.

Output: Per-token or per-word predictions from a linear probing classifier applied to the model's hidden states.

Scoring recipe

for layer in model_layers:
    X, y = extract_hidden_states_and_labels(layer, dataset)
    probe = LinearClassifier(reg='elasticnet', loss='categorical_crossentropy')
    probe.fit(X_train, y_train, epochs=10, batch_size=512)
    preds = probe.predict(X_test)
    acc = accuracy_score(y_test, preds)
    selectivity = acc / accuracy_random_baseline

Common pitfalls

• Using average pooling over subword tokens instead of the specified last-activation representation.
• Interpreting downstream fine-tuning accuracy as the evaluation metric; the protocol specifically measures linguistic knowledge via layer-wise probing.
• Failing to compute selectivity, which risks attributing probe memorization to actual representation quality.

Evidence (verbatim from paper)

We evaluated our method on 3 linguistic tasks: POS tagging using the Penn TreeBank (Marcus et al., 1993), syntactic chunking using CoNLL 2000 shared task dataset (Tjong Kim Sang and Buchholz, 2000), and semantic tagging using the Parallel Meaning Bank data (Abzianidze et al., 2017). We used standard splits for training, development and test data. Classifier Settings: We used a linear probing classifier with elastic-net regularization, using a categorical cross-entropy loss, optimized by Adam (Kingma and Ba, 2014). Training is run with shuffled mini-batches of size 512 and stopped after 10 epochs. The regularization weights are trained using grid-search. For sub-word based models, we use the last activation value to be the representative of the word following Durrani et al. (2019). We computed selectivity (Hewitt and Liang, 2019) to ensure that our results reflect the property of representations and not the probe's capacity to memorize.

Citation

@misc{durrani2021transfer,
  title={How transfer learning impacts linguistic knowledge in deep NLP models?},
  author={Durrani et al. (2021)},
  year={2021},
  note={arXiv:2105.15179}
}

• arXiv: 2105.15179