# Linguistic Probing Eval

> Evaluates how fine-tuning on downstream NLP tasks redistributes linguistic knowledge across transformer layers. It probes for part-of-speech tagging, syntactic chunking, and semantic tagging capabilities using linear classifiers on layer-wise hidden states. Use when the user wants to benchmark on Penn TreeBank, CoNLL 2000, Parallel Meaning Bank, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/linguistic-probing-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/linguistic-probing-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/linguistic-probing-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/linguistic-probing-eval

---


# linguistic-probing-eval

> How transfer learning impacts linguistic knowledge in deep NLP models? — Durrani et al. (2021) (arXiv:2105.15179, 2021)

## What this evaluates

Evaluates how fine-tuning on downstream NLP tasks redistributes linguistic knowledge across transformer layers. It probes for part-of-speech tagging, syntactic chunking, and semantic tagging capabilities using linear classifiers on layer-wise hidden states.

## Datasets

- **Penn TreeBank** — total ?; splits: train (-1), val (-1), test (-1)
- **CoNLL 2000** — total ?; splits: train (-1), val (-1), test (-1)
- **Parallel Meaning Bank** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy computed on the test split for each linguistic task.
- `selectivity` — range: [0, 1]
  - Ratio of the probing classifier's accuracy to a random baseline accuracy, used to control for probe memorization.

## Input / output format

**Input**: Tokenized word/subword sequences with corresponding linguistic labels (POS tags, chunk tags, or semantic tags) from the respective datasets.

**Output**: Per-token or per-word predictions from a linear probing classifier applied to the model's hidden states.

## Scoring recipe

```python
for layer in model_layers:
    X, y = extract_hidden_states_and_labels(layer, dataset)
    probe = LinearClassifier(reg='elasticnet', loss='categorical_crossentropy')
    probe.fit(X_train, y_train, epochs=10, batch_size=512)
    preds = probe.predict(X_test)
    acc = accuracy_score(y_test, preds)
    selectivity = acc / accuracy_random_baseline
```

## Common pitfalls

- Using average pooling over subword tokens instead of the specified last-activation representation.
- Interpreting downstream fine-tuning accuracy as the evaluation metric; the protocol specifically measures linguistic knowledge via layer-wise probing.
- Failing to compute selectivity, which risks attributing probe memorization to actual representation quality.

## Evidence (verbatim from paper)

> We evaluated our method on 3 linguistic tasks: POS tagging using the Penn TreeBank (Marcus et al., 1993), syntactic chunking using CoNLL 2000 shared task dataset (Tjong Kim Sang and Buchholz, 2000), and semantic tagging using the Parallel Meaning Bank data (Abzianidze et al., 2017). We used standard splits for training, development and test data. Classifier Settings: We used a linear probing classifier with elastic-net regularization, using a categorical cross-entropy loss, optimized by Adam (Kingma and Ba, 2014). Training is run with shuffled mini-batches of size 512 and stopped after 10 epochs. The regularization weights are trained using grid-search. For sub-word based models, we use the last activation value to be the representative of the word following Durrani et al. (2019). We computed selectivity (Hewitt and Liang, 2019) to ensure that our results reflect the property of representations and not the probe's capacity to memorize.

## Citation

```bibtex
@misc{durrani2021transfer,
  title={How transfer learning impacts linguistic knowledge in deep NLP models?},
  author={Durrani et al. (2021)},
  year={2021},
  note={arXiv:2105.15179}
}
```

- arXiv: 2105.15179