# Natural Language Processing

> NLP text processing and analysis

- Skill: `ffsshhttiikk/natural-language-processing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ffsshhttiikk/natural-language-processing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ffsshhttiikk/natural-language-processing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: ffsshhttiikk (https://skillmd.com/u/ffsshhttiikk)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ffsshhttiikk/natural-language-processing

---


## What I do

- Process and analyze text data
- Build text classification models
- Extract entities and features from text
- Implement language models
- Create text embeddings
- Build language understanding systems
- Work with multilingual text

## When to use me

Use me when:
- Building text classification systems
- Extracting information from documents
- Creating chatbots or virtual assistants
- Analyzing sentiment in text
- Working with large language models
- Processing multilingual data

## Key Concepts

### Text Preprocessing
```python
import re
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

def preprocess_text(text):
    # Lowercase
    text = text.lower()
    
    # Remove URLs
    text = re.sub(r'http\S+|www\S+', '', text)
    
    # Remove special chars, keep letters
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    
    # Tokenize
    tokens = word_tokenize(text)
    
    # Remove stopwords
    stop_words = set(stopwords.words('english'))
    tokens = [t for t in tokens if t not in stop_words]
    
    # Lemmatize
    lemmatizer = WordNetLemmatizer()
    tokens = [lemmatizer.lemmatize(t) for t in tokens]
    
    return ' '.join(tokens)

# Apply
df['processed_text'] = df['raw_text'].apply(preprocess_text)
```

### Text Vectorization
```python
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

# TF-IDF
tfidf = TfidfVectorizer(max_features=5000, ngram_range=(1,2))
X_tfidf = tfidf.fit_transform(texts)

# Count (for LDA)
count_vec = CountVectorizer(max_features=5000)
X_count = count_vec.fit_transform(texts)

# LDA for topic modeling
lda = LatentDirichletAllocation(n_components=10)
topics = lda.fit_transform(X_count)
```

### Transformer Models
```python
from transformers import pipeline, AutoTokenizer, AutoModel

# Pre-trained sentiment
sentiment = pipeline("sentiment-analysis")
result = sentiment("This product is amazing!")

# Named Entity Recognition
ner = pipeline("ner", entity="ORGANIZATION")
entities = ner("Apple Inc. is headquartered in Cupertino")

# Text generation
generator = pipeline("text-generation", model="gpt2")
output = generator("Once upon a time", max_length=50)
```

### Common NLP Tasks
- **Classification**: Spam detection, sentiment
- **NER**: Named entity extraction
- **Translation**: Language conversion
- **Summarization**: Abstractive/Extractive
- **QA**: Question answering
- **Chatbots**: Conversational AI