Nlp

nlp — tokenize the text, pick the model type, pick the metric

You own the language-modeling discipline: how raw text becomes tokens, which transformer architecture fits a task, and which metric actually tells you whether it worked. When the question is "which tokenizer," "BERT or GPT or T5 for this," "why does my Catalan text cost 3× the tokens," or "is this BLEU score meaningful," this is the skill. You stop at retrieval, the RAG loop, prompt wording, and the training step itself — those route out (below).

Route out first (loud — do not duplicate these)

• Retrieval embeddings + vector search (which embedding model, chunking, recall@k, rerank) → ../embeddings-search/SKILL.md. Sentence embeddings live here as a task; using them to retrieve is theirs.
• The retrieve → prompt → generate → answer loop and groundedness → ../rag/SKILL.md.
• Prompt wording / few-shot / system prompts → ../prompt-engineering/SKILL.md.
• Training the network (LoRA/SFT, trainer loop, PyTorch) → ../finetuning/SKILL.md
  • ../deep-learning/SKILL.md.
• The training corpus itself (JSONL messages, label sets) → ../training-data/SKILL.md.

Decision: model type per task (get this right before anything else)

Pick the architecture from the task's shape, not from what is trendy. A decoder LLM can technically classify, but a fine-tuned encoder is smaller, faster, cheaper, and usually more accurate on a fixed-label task.

* Architecture families are stable; specific checkpoints and their licenses are not — check the HF model card before you commit (licenses change; see open-weights). ModernBERT (2024) is a current long-context encoder; verify the latest at author time.

The two most common own-goals: reaching for a 7B decoder to do sentiment on 5 classes (an encoder does it for a fraction of the cost), and forcing an encoder to generate (it cannot — it has no decoder).

1. Tokenization

Every downstream number depends on this step, and its failures are silent. The single load-bearing rule:

Load the tokenizer that shipped with the checkpoint, and use the same one at train and inference. AutoTokenizer.from_pretrained(same_checkpoint). A train/inference tokenizer mismatch — different vocab, different special tokens, different casing/normalization — maps text to token ids the model never saw and corrupts everything downstream with no error.

from transformers import AutoTokenizer   # transformers current major ~v5 (verify at author time)

tok = AutoTokenizer.from_pretrained("bert-base-cased")
enc = tok("Tokenizers matter.", return_offsets_mapping=True)
tok.convert_ids_to_tokens(enc["input_ids"])
# ['[CLS]', 'Token', '##izers', 'matter', '.', '[SEP]']  — note WordPiece '##' continuation + added specials

The four algorithms (full mechanics in references/tokenization.md):

• Byte-level BPE has no [UNK]. Base vocab is exactly 256 (all byte values), so every emoji, accent, and script maps to some byte sequence — nothing falls out as unknown (verified: HF NLP course ch.6). WordPiece/word-level tokenizers do have [UNK] and lose OOV content.
• SentencePiece is reversible — it treats space as a normal symbol (the ▁ meta-symbol), so decode(encode(x)) == x without language-specific detokenization rules. That is why it dominates multilingual models.

Special tokens are not decoration. [CLS]/<s> carries the pooled sentence representation for classification; [SEP]/</s> marks segment/end; [PAD] fills a batch (and must be masked out via attention_mask); [MASK] is the MLM target; [UNK] is the fallback. Names differ by model ([CLS] in BERT vs <s> in RoBERTa) — another reason to never hand-roll the tokenizer.

Why it matters — three concrete costs:

• $ cost & context. Token count is the bill and the context budget. Fewer tokens per sentence = cheaper calls and more room in the window.
• OOV / information loss. A tokenizer that emits [UNK] throws away content it can't represent; byte-level/SentencePiece degrade gracefully instead.
• Fairness. Vocab trained mostly on English fragments other scripts far harder — the same meaning costs more tokens, more money, and more latency (section 5).

2. Tasks

Text classification (encoder + classification head)

from transformers import pipeline
clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
clf("The service was slow but the food was incredible.")
# [{'label': 'POSITIVE', 'score': 0.99...}]

Metric: accuracy on balanced data; macro-F1 the moment classes are imbalanced (accuracy lies when 95% of rows are one class).

Token classification / NER (encoder, per-token labels)

Labels are B-/I-/O spans aligned to subword tokens: the first subword of a word gets the label, continuation subwords and special tokens get -100 (ignored by the loss). Evaluate with seqeval at the entity level, never per-token accuracy (per-token accuracy is inflated by the flood of O tokens).

from transformers import pipeline
ner = pipeline("token-classification", aggregation_strategy="simple")
ner("Ada Lovelace worked in London.")
# groups subwords back into entities: PER 'Ada Lovelace', LOC 'London'

Seq2seq — summarization / translation (encoder-decoder)

summ = pipeline("summarization", model="facebook/bart-large-cnn")
summ(long_article, max_length=130, min_length=30)

Metric: ROUGE for summarization, BLEU/chrF for translation — with the heavy caveat in section 4.

Sentence embeddings (SBERT — the task, not retrieval)

from sentence_transformers import SentenceTransformer   # sentence-transformers ~v5 (verify)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
emb = model.encode(["The weather is lovely today.", "It's so sunny outside!"])
model.similarity(emb, emb)   # semantic textual similarity / clustering / paraphrase mining

Producing/judging embeddings for retrieval (model choice, chunking, recall@k, rerankers) is embeddings-search, not here.

3. Evaluation — pick the metric that matches the task

The caveat that governs this whole section: BLEU, ROUGE, and chrF are n-gram/character overlap metrics and correlate weakly with human judgment on open-ended and creative generation — they reward matching the reference's exact phrasing, so a correct paraphrase scores low and a fluent-but-wrong copy scores high (well documented; e.g. the summarization and MT-evaluation literature). Use them for regression tracking on a fixed reference set, never as the final verdict on quality. For open-ended output, use an LLM-as-judge rubric plus a human spot-check — and know the judge has its own biases (position, verbosity, self-preference), so pin the rubric and randomize order.

Perplexity = exp(mean token NLL): lower means the model predicts held-out text better. It is tokenizer-dependent, so two models with different tokenizers have non-comparable perplexities — only compare within the same tokenizer/vocab. Runnable snippets for seqeval, sacrebleu, ROUGE, perplexity, and an LLM-judge harness are in references/evaluation.md.

4. Multilingual pitfalls

The English-centric trap: a tokenizer whose vocab was learned mostly on English over-fragments other scripts. The same sentence in Ukrainian, Arabic, Hindi, or even accented Catalan can take 2–15× more tokens than its English equivalent (Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023). That "fertility" (tokens per word) inflation is a triple tax:

• Money — more tokens per identical meaning = a proportionally larger bill for the same work.
• Context — over-fragmented text eats the window faster, so fewer few-shot examples fit and long documents truncate sooner.
• Quality — sequences fragmented into byte-shards are harder to model, degrading accuracy for exactly the users the tool already serves worst.

Mitigations: prefer a multilingual tokenizer/model (mT5, XLM-R, a SentencePiece-based model) whose vocab actually covers your languages; measure fertility on your own corpus (tokens per word, per language) before you commit; and don't benchmark cost or latency only on English.

Guardrails / gotchas

• Tokenizer must match the checkpoint, at train and inference. Mismatch corrupts silently, no error. The most expensive bug in this skill.
• Encoders can't generate; don't classify with a giant decoder by default. Match architecture to task shape.
• BLEU/ROUGE/chrF ≠ quality on open-ended text. Overlap metrics; weak human correlation. Track regressions with them; judge quality with an LLM-judge + human.
• Entity-F1 (seqeval), not token accuracy, for NER. O tokens inflate accuracy toward 1.0.
• macro-F1, not accuracy, on imbalanced classes.
• Perplexity is tokenizer-relative — never compare it across different tokenizers.
• Don't benchmark tokenization/cost only in English — fertility varies 2–15× across scripts.
• Never assert a model's license from memory — check the current model card; license classes shift (Llama = Meta Community license, not OSI-open; Gemma = custom terms; etc.).

Related skills

• embeddings-search — retrieval embeddings, chunking, recall@k, reranking. NLP owns making/judging sentence embeddings as a task; using them to search is theirs.
• rag — the full retrieve→generate answer loop and groundedness.
• prompt-engineering — the wording of the prompt.
• finetuning + deep-learning — actually training/adapting the network (this skill picks the type and metric; those move the weights).
• training-data — building the labeled corpus you train on.

Checklist

• Architecture picked from task shape (encoder / decoder / enc-dec), not habit.
• Tokenizer loaded from the same checkpoint, used identically at train and inference.
• Tokenizer choice justified vs OOV, cost, and — if multilingual — measured fertility.
• Special tokens and attention_mask handled (padding masked, -100 on ignored labels).
• Metric matches the task: macro-F1 (imbalanced), entity-F1/seqeval (NER), ROUGE/BLEU/chrF only as a regression signal, LLM-judge + human for open-ended.
• Perplexity compared only within one tokenizer.
• Retrieval / RAG / prompt / training concerns routed to the sibling skill, not re-solved here.

References

• references/tokenization.md — BPE/WordPiece/Unigram/byte-level training mechanics, special tokens per family, offset mapping, and a fertility-measuring snippet.
• references/evaluation.md — runnable seqeval, sacrebleu (BLEU/chrF), ROUGE, perplexity, and an LLM-as-judge rubric, with when each lies.