# Ds Topic Modeling > "Implements topic modeling using Latent Dirichlet Allocation (LDA) Non-negative Matrix Factorization (NMF), and other topic extraction methods" - Skill: `paulpas/ds-topic-modeling` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/ds-topic-modeling` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/ds-topic-modeling/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/ds-topic-modeling --- # Topic Modeling Comprehensive guide to topic modeling in machine learning and data science workflows. ## When to Use This Skill - Solving real-world unsupervised learning problems - Building machine learning pipelines with topic modeling - Implementing best practices for topic modeling - Optimizing model performance using topic modeling techniques - Learning industry-standard approaches to topic modeling ## When NOT to Use This Skill - When using pre-built libraries without understanding underlying concepts - For toy problems that don't require topic modeling rigor - When domain expertise in specific problem requires different approach - If your problem doesn't require the complexity this skill provides ## Purpose and Key Concepts Topic Modeling is a critical component of the machine learning workflow. This skill covers: 1. **Theoretical foundations** — Mathematical principles and statistical concepts 2. **Practical implementation** — Working code examples and patterns 3. **Common pitfalls** — Mistakes to avoid and how to recover from them 4. **Best practices** — Industry-standard approaches and optimization techniques ## Core Workflow 1. **Understand the problem** — Clearly define what you're solving for 2. **Select approach** — Choose the right technique for your data and constraints 3. **Implement solution** — Write clean, tested code following best practices 4. **Validate results** — Verify your implementation with tests and validation 5. **Optimize performance** — Improve efficiency and accuracy incrementally ## Implementation Patterns ### Pattern 1: Basic Topic Modeling ```python import pandas as pd import numpy as np from sklearn.feature_extraction.text import CountVectorizer from sklearn.decomposition import LatentDirichletAllocation from typing import List, Dict, Any def basic_topic_modeling(texts: List[str], n_topics: int = 5) -> Dict[str, Any]: """ Perform basic topic modeling using Latent Dirichlet Allocation (LDA). Args: texts: List of raw text documents n_topics: Number of topics to extract Returns: Dictionary containing fitted model, vectorizer, and topic-word distributions """ if not texts: raise ValueError("Input texts list cannot be empty") # Vectorize text documents vectorizer = CountVectorizer(max_df=0.95, min_df=2, stop_words='english') doc_term_matrix = vectorizer.fit_transform(texts) # Initialize and fit LDA model lda_model = LatentDirichletAllocation( n_components=n_topics max_iter=10 learning_method='online' random_state=42 ) lda_model.fit(doc_term_matrix) # Extract top words per topic feature_names = vectorizer.get_feature_names_out() topics = {} for idx, topic in enumerate(lda_model.components_): top_words_idx = topic.argsort()[:-10:-1] topics[f'topic_{idx}'] = [feature_names[i] for i in top_words_idx] return { 'model': lda_model 'vectorizer': vectorizer 'topics': topics 'doc_term_matrix': doc_term_matrix } ``` ### Pattern 2: Production-Ready Topic Modeling ```python import logging import pandas as pd import numpy as np from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.decomposition import NMF from sklearn.pipeline import Pipeline from typing import Any, Dict, List, Optional import warnings warnings.filterwarnings('ignore') logger = logging.getLogger(__name__) class TopicModeling: """Production-grade implementation of Topic Modeling using NMF""" def __init__(self, n_topics: int = 5, max_features: int = 1000, random_state: int = 42): self.n_topics = n_topics self.max_features = max_features self.random_state = random_state self.pipeline: Optional[Pipeline] = None self.feature_names: List[str] = [] def _validate_input(self, data: pd.DataFrame, text_column: str) -> None: if text_column not in data.columns: raise ValueError(f"Column '{text_column}' not found in DataFrame") if data[text_column].isnull().any(): logger.warning("Dropping rows with missing text data") data = data.dropna(subset=[text_column]) if len(data) < self.n_topics: raise ValueError("Dataset size must be greater than n_topics") def _build_pipeline(self) -> Pipeline: vectorizer = TfidfVectorizer( max_features=self.max_features stop_words='english' ngram_range=(1, 2) ) nmf_model = NMF( n_components=self.n_topics init='nndsvd' random_state=self.random_state max_iter=200 ) return Pipeline([('tfidf', vectorizer), ('nmf', nmf_model)]) def execute(self, data: pd.DataFrame, text_column: str = 'text') -> Dict[str, Any]: """Execute Topic Modeling on data""" self._validate_input(data, text_column) self.pipeline = self._build_pipeline() tfidf_matrix = self.pipeline.named_steps['tfidf'].fit_transform(data[text_column]) self.pipeline.fit(tfidf_matrix) feature_names = self.pipeline.named_steps['tfidf'].get_feature_names_out() topic_words = {} for i, topic in enumerate(self.pipeline.named_steps['nmf'].components_): top_indices = topic.argsort()[:-10:-1] topic_words[f'topic_{i}'] = [feature_names[idx] for idx in top_indices] reconstructed = self.pipeline.named_steps['nmf'].transform(tfidf_matrix) @ \ self.pipeline.named_steps['nmf'].components_ inertia = float(np.linalg.norm(tfidf_matrix.toarray() - reconstructed)) return { 'topics': topic_words 'inertia': inertia 'pipeline': self.pipeline 'document_topics': self.pipeline.named_steps['nmf'].transform(tfidf_matrix) } ``` ## Best Practices - ✅ Always validate your implementation on test data - ✅ Document your assumptions and methodology - ✅ Use version control for reproducibility - ✅ Monitor performance metrics in production - ✅ Periodically review and update your approach - ✅ Test with edge cases and outliers - ✅ Log all significant operations for debugging - Follow DRY (Don't Repeat Yourself) and KISS (Keep It Simple, Stupid) principles for maintainable code ## Common Pitfalls | Pitfall | Problem | Solution | | --- --- ## Constraints ### MUST DO - Validate all data preprocessing steps are fit-only on training data, never on validation or test sets - Implement reproducible pipelines with fixed random seeds and deterministic operations where possible - Report model performance with confidence intervals via bootstrapping or cross-validation across multiple runs - Log all experiments with parameters, metrics, and artifacts using MLflow or equivalent tracking system ### MUST NOT DO - Do not evaluate a model on the same data used for training — always hold out a proper test set - Avoid overfitting to the validation set by limiting hyperparameter search iterations - Never use features that can only be computed at inference time (look-ahead bias) - Do not report single-run accuracy without statistical significance testing or error bars ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Topic Modeling — Wikipedia](https://en.wikipedia.org/wiki/Topic_model) - [Gensim Documentation](https://radimrehurek.com/gensim/) - [LDA Topic Modeling — Scikit-learn docs](https://scikit-learn.org/stable/modules/lda_lda.html) - [BERTopic Documentation](https://maartengr.github.io/BERTopic/) - [NMF for Topic Extraction (Scikit-learn)](https://scikit-learn.org/stable/modules/decomposition.html#nmf)