# Genomic LLM Embeddings

> Build DNA embeddings via k-mer frequency vectors or genomic LMs (Nucleotide Transformer, DNABERT-2, HyenaDNA). Use when embedding DNA for ML, choosing k-mer/BPE tokenization, or probing embedding quality.

- Skill: `pavel-kravchenko/genomic-llm-embeddings` (Agent Skill)
- Install (CLI): `npx skillmds@latest add pavel-kravchenko/genomic-llm-embeddings`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pavel-kravchenko/genomic-llm-embeddings/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: pavel-kravchenko (https://skillmd.com/u/pavel-kravchenko)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/pavel-kravchenko/genomic-llm-embeddings

---


# genomic-llm-embeddings

## When to Use

- Turning raw DNA sequences into fixed-length numeric vectors for a downstream classifier or clustering task.
- Deciding between character-level, k-mer, or BPE tokenization before fine-tuning a genomic LM.
- Sanity-checking a new sequence embedding (from a pretrained model or your own encoder) against a cheap k-mer baseline before trusting it.
- Choosing between short-context transformers (DNABERT-2, Nucleotide Transformer) and long-context models (HyenaDNA, Evo) based on how far apart regulatory elements are.
- Building a synthetic probe task (e.g., promoter motif detection) to validate an embedding pipeline end-to-end.

## Version Compatibility

- Python ≥3.10, NumPy ≥1.24
- transformers ≥4.40 (only if loading pretrained genomic LMs)
- Reference models: Nucleotide Transformer v2 (InstaDeepAI, 500M-multi-species), DNABERT-2 (117M), HyenaDNA (up to 1M bp context), Evo (prokaryotic/viral, StripedHyena backbone)

## Prerequisites

- `pip install numpy` (add `transformers torch` only for loading real pretrained models)
- Familiarity with basic sequence handling (`biopython` skill) helps but is not required
- Related skill: `genomic-foundation-models` for fine-tuning details of the models named above

## Tokenization Strategy Selection

| Strategy | Context | Strength | Limitation |
|---|---:|---|---|
| Character (A/C/G/T/N) | Long | Max resolution | Long token sequences, slow attention |
| k-mer (k=3..6) | Short/medium | Fast, interpretable baseline | Loses positional/order info; k=6 already gives 4096-word vocab |
| DNABERT-2 (BPE) | ~512 bp windows | Strong short-window tasks | Limited context length |
| Nucleotide Transformer (6-mer, stride 1) | kb-scale | Good transfer embeddings | ~L/6 tokens, higher memory than k-mer counts |
| HyenaDNA (character-level, SSM) | up to 1M bp | Captures distal regulatory interactions | Heavier training/inference than short-context models |
| Evo (character-level, StripedHyena) | genome-scale | Prokaryotic/viral generation & scoring | Not trained on mammalian genomes — don't use for human regulatory tasks |

**Goal:** Turn a DNA sequence into a fixed-length vector without training anything.
**Approach:** Slide a window of length `k` across the sequence, count k-mer occurrences, and normalize into a frequency vector over the fixed `4**k` vocabulary. This is the standard sanity-check baseline before reaching for a pretrained model.

```python
import numpy as np
from collections import Counter

def kmers(seq: str, k: int = 6) -> list[str]:
    """Slide a window of length k across seq, returning overlapping k-mers."""
    seq = seq.upper()
    return [seq[i:i + k] for i in range(len(seq) - k + 1)]

def kmer_embedding(seq: str, vocab: list[str], k: int = 3) -> np.ndarray:
    """Normalized k-mer frequency vector over a fixed vocabulary (order-independent)."""
    tokens = kmers(seq, k)
    counts = Counter(tokens)
    vec = np.array([counts[v] for v in vocab], dtype=float)
    return vec / (vec.sum() + 1e-9)

alphabet = ["A", "C", "G", "T"]
vocab_3 = [a + b + c for a in alphabet for b in alphabet for c in alphabet]  # 64-dim

example_vec = kmer_embedding("ATGATGATGCCC", vocab_3, k=3)
assert example_vec.shape[0] == 64
assert abs(example_vec.sum() - 1.0) < 1e-6
```

**Goal:** Check whether an embedding (k-mer or model-derived) actually separates two classes before trusting it downstream.
**Approach:** Train-free nearest-centroid probe — compute per-class centroids on train embeddings, classify test embeddings by nearest centroid. Cheap, has no hyperparameters, and exposes garbage embeddings immediately.

```python
import numpy as np

def nearest_centroid_predict(X_train: np.ndarray, y_train: np.ndarray, X_test: np.ndarray) -> np.ndarray:
    """Binary nearest-centroid classifier: assign each test row to the closer class mean."""
    c0 = X_train[y_train == 0].mean(axis=0)
    c1 = X_train[y_train == 1].mean(axis=0)
    d0 = ((X_test - c0) ** 2).sum(axis=1)
    d1 = ((X_test - c1) ** 2).sum(axis=1)
    return (d1 < d0).astype(int)

def random_dna(n: int, rng: np.random.Generator) -> str:
    """Generate a random ACGT sequence of length n."""
    return "".join(rng.choice(list("ACGT"), size=n))

def inject_motif(seq: str, motif: str, pos: int) -> str:
    """Splice a motif into seq at position pos (overwrites in place, keeps length)."""
    return seq[:pos] + motif + seq[pos + len(motif):]

def demo():
    """Synthetic promoter-vs-background probe: TATA box injected at pos 20 in half the sequences."""
    rng = np.random.default_rng(7)
    n_samples, length, motif = 120, 80, "TATAAA"
    seqs, labels = [], []
    for _ in range(n_samples):
        s = random_dna(length, rng)
        if rng.random() < 0.5:
            s = inject_motif(s, motif, pos=20)
            labels.append(1)
        else:
            labels.append(0)
        seqs.append(s)

    X = np.stack([kmer_embedding(s, vocab_3, k=3) for s in seqs])
    y = np.array(labels)
    X_train, y_train = X[:90], y[:90]
    X_test, y_test = X[90:], y[90:]

    pred = nearest_centroid_predict(X_train, y_train, X_test)
    acc = (pred == y_test).mean()
    assert acc > 0.8, f"probe accuracy too low: {acc}"  # motif is easy; k-mer baseline should nail it
    print(f"nearest-centroid probe accuracy: {acc:.3f}")

if __name__ == "__main__":
    demo()
```

**Goal:** Get real embeddings from a pretrained genomic LM instead of a k-mer baseline.
**Approach:** Use `transformers` to load a genomic foundation model, tokenize, and mean-pool the last hidden state into a single vector per sequence.

```python
def embed_with_nucleotide_transformer(seqs: list[str]) -> "np.ndarray":
    """Mean-pooled embeddings from Nucleotide Transformer v2 (500M-multi-species).
    Requires: pip install transformers torch
    """
    import torch
    from transformers import AutoTokenizer, AutoModel

    model_name = "InstaDeepAI/nucleotide-transformer-v2-500m-multi-species"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModel.from_pretrained(model_name).eval()

    tokens = tokenizer(seqs, return_tensors="pt", padding=True, truncation=True)
    with torch.no_grad():
        out = model(**tokens)
    mask = tokens["attention_mask"].unsqueeze(-1)
    pooled = (out.last_hidden_state * mask).sum(1) / mask.sum(1)
    return pooled.numpy()
```

## Pitfalls

- Mixing tokenization schemes between train and inference silently destroys performance — a model trained on 6-mer tokens fed character-level input at inference will not error, just degrade quietly.
- Always compare against a k-mer baseline first; without one, overfitting or a broken embedding pipeline is very hard to detect.
- k-mer vocab explodes with k: 4^6 = 4096, 4^8 = 65,536 — keep explicit-count k-mers to k≤6, use BPE or learned embeddings beyond that.
- Match sequence window length across models being compared — truncating to a short window can flip labels that depend on distal motifs (see the `distal_interaction_label` pattern: a motif near the end is silently dropped if you truncate to 200bp).
- Evo is trained on prokaryotic/viral genomes only — do not use it for human/mammalian regulatory tasks.
- Nearest-centroid probes only validate that classes separate at all; a low probe score means "fix the embedding," not "add a bigger downstream model."

## See Also

- `genomic-foundation-models` — fine-tuning and inference details for NT, DNABERT-2, HyenaDNA, Evo
- `protein-language-models` — ESM2 embeddings for protein sequences (analogous workflow for proteins)
- `ai-science-epigenomic-sequence-models` — regulatory-activity prediction (Enformer/AlphaGenome) from DNA
- `bio-sequence-manipulation-motif-search` — motif scanning utilities used alongside embedding probes