Data Mining

Data Mining

Extract valuable patterns, relationships, and insights from large datasets using computational algorithms.

Overview

Data mining is the process of discovering patterns, correlations, and anomalies within large datasets to predict outcomes and extract actionable knowledge. It combines techniques from statistics, machine learning, and database systems to analyze data from multiple perspectives. This skill covers association rule learning, clustering algorithms, classification methods, and anomaly detection techniques.

Core Data Mining Techniques

Association Rule Learning

Key Concepts

Support: Frequency of itemset in dataset

• Formula: support(X) = count(X) / total_transactions
• High support = common itemset

Confidence: Strength of rule X → Y

• Formula: confidence(X → Y) = support(X ∪ Y) / support(X)
• High confidence = strong predictive relationship

Lift: Correlation between X and Y

• Formula: lift(X → Y) = confidence(X → Y) / support(Y)
• Lift > 1: positive correlation
• Lift < 1: negative correlation
• Lift = 1: independence

Apriori Algorithm

from mlxtend.frequent_patterns import apriori, association_rules
import pandas as pd

# Prepare transaction data (one-hot encoded)
# Each row is a transaction, each column is an item (True/False)
transactions = pd.DataFrame({
    'milk': [True, True, False, True],
    'bread': [True, True, True, False],
    'butter': [False, True, True, True],
    'beer': [False, False, True, True]
})

# Find frequent itemsets
frequent_itemsets = apriori(transactions, min_support=0.5, use_colnames=True)

# Generate association rules
rules = association_rules(frequent_itemsets, metric="confidence", min_threshold=0.7)

# Filter by lift
strong_rules = rules[rules['lift'] > 1.2]

FP-Growth Algorithm

Faster than Apriori for large datasets:

• Builds FP-tree (compressed representation)
• Mines frequent patterns without candidate generation
• More efficient for dense datasets

Applications

1. Market Basket Analysis

   • Product placement optimization
   • Cross-selling strategies
   • Bundle pricing

2. Web Usage Mining

   • Page navigation patterns
   • User behavior analysis
   • Content recommendation

3. Healthcare

   • Symptom-disease associations
   • Drug interaction detection
   • Treatment effectiveness patterns

Clustering Algorithms

K-Means Clustering

Algorithm:

1. Initialize K cluster centroids randomly
2. Assign each point to nearest centroid
3. Recalculate centroids as mean of assigned points
4. Repeat until convergence

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

# Fit K-Means
kmeans = KMeans(n_clusters=3, random_state=42)
clusters = kmeans.fit_predict(X)

# Visualize
plt.scatter(X[:, 0], X[:, 1], c=clusters, cmap='viridis')
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], 
            marker='x', s=200, linewidths=3, color='r')
plt.show()

# Elbow method for optimal K
inertias = []
for k in range(1, 11):
    kmeans = KMeans(n_clusters=k, random_state=42)
    kmeans.fit(X)
    inertias.append(kmeans.inertia_)

plt.plot(range(1, 11), inertias, marker='o')
plt.xlabel('Number of Clusters')
plt.ylabel('Inertia')
plt.title('Elbow Method')
plt.show()

Strengths: Fast, scalable, simple Weaknesses: Requires K specification, sensitive to outliers, assumes spherical clusters

DBSCAN (Density-Based Spatial Clustering)

Parameters:

• eps: Maximum distance between points in same cluster
• min_samples: Minimum points to form dense region

from sklearn.cluster import DBSCAN

# Fit DBSCAN
dbscan = DBSCAN(eps=0.5, min_samples=5)
clusters = dbscan.fit_predict(X)

# -1 indicates noise/outliers
n_clusters = len(set(clusters)) - (1 if -1 in clusters else 0)
n_noise = list(clusters).count(-1)

Strengths: Finds arbitrary-shaped clusters, identifies outliers, no need to specify K Weaknesses: Sensitive to parameters, struggles with varying densities

Hierarchical Clustering

Agglomerative (bottom-up):

1. Start with each point as cluster
2. Merge closest clusters
3. Repeat until single cluster

from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage

# Fit hierarchical clustering
hierarchical = AgglomerativeClustering(n_clusters=3, linkage='ward')
clusters = hierarchical.fit_predict(X)

# Create dendrogram
linkage_matrix = linkage(X, method='ward')
dendrogram(linkage_matrix)
plt.title('Hierarchical Clustering Dendrogram')
plt.show()

Linkage methods:

• Ward: Minimizes within-cluster variance
• Complete: Maximum distance between clusters
• Average: Average distance between clusters
• Single: Minimum distance between clusters

Classification Techniques

Decision Trees

from sklearn.tree import DecisionTreeClassifier, plot_tree

# Train decision tree
dt = DecisionTreeClassifier(max_depth=5, min_samples_split=20, random_state=42)
dt.fit(X_train, y_train)

# Visualize tree
plt.figure(figsize=(20, 10))
plot_tree(dt, feature_names=feature_names, class_names=class_names, filled=True)
plt.show()

# Feature importance
importances = pd.DataFrame({
    'feature': feature_names,
    'importance': dt.feature_importances_
}).sort_values('importance', ascending=False)

Random Forest

Ensemble of decision trees:

• Reduces overfitting
• Improves accuracy
• Provides feature importance

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100, max_depth=10, random_state=42)
rf.fit(X_train, y_train)

# Out-of-bag score (internal validation)
rf_oob = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)
rf_oob.fit(X_train, y_train)
print(f"OOB Score: {rf_oob.oob_score_}")

Anomaly Detection

Isolation Forest

Isolates anomalies by randomly partitioning data:

• Anomalies are easier to isolate (fewer splits needed)
• Normal points require more splits

from sklearn.ensemble import IsolationForest

# Fit Isolation Forest
iso_forest = IsolationForest(contamination=0.1, random_state=42)
predictions = iso_forest.fit_predict(X)

# -1 indicates anomaly, 1 indicates normal
anomalies = X[predictions == -1]
normal = X[predictions == 1]

Local Outlier Factor (LOF)

Measures local density deviation:

• Compares density of point to neighbors
• Low density relative to neighbors = outlier

from sklearn.neighbors import LocalOutlierFactor

lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
predictions = lof.fit_predict(X)

Data Mining Workflow

1. Business Understanding

• Define objectives
• Identify success criteria
• Assess resources and constraints

2. Data Understanding

• Collect initial data
• Explore data characteristics
• Verify data quality

3. Data Preparation

• Select relevant data
• Clean data (handle missing values, outliers)
• Transform features
• Integrate data from multiple sources

4. Modeling

• Select modeling techniques
• Build models
• Tune hyperparameters

5. Evaluation

• Assess model performance
• Validate against business objectives
• Determine next steps

6. Deployment

• Plan deployment strategy
• Implement model in production
• Monitor and maintain

Best Practices

Data Preparation

1. Handle missing values appropriately

   • Imputation strategies
   • Consider missingness patterns
   • Document decisions

2. Scale features when necessary

   • Distance-based algorithms (K-Means, DBSCAN) require scaling
   • Tree-based algorithms don't require scaling

3. Encode categorical variables

   • One-hot encoding for nominal categories
   • Ordinal encoding for ordered categories

Model Selection

1. Match algorithm to problem type

   • Association rules for relationship discovery
   • Clustering for segmentation
   • Classification for prediction

2. Consider interpretability vs. accuracy trade-off

   • Decision trees: interpretable
   • Random forests: more accurate, less interpretable

3. Validate thoroughly

   • Cross-validation
   • Hold-out test set
   • Domain expert review

Performance Optimization

1. Use appropriate data structures

   • Sparse matrices for high-dimensional data
   • Efficient storage formats

2. Leverage parallel processing

   • Many algorithms support n_jobs parameter
   • Distribute computation across cores

3. Sample large datasets

   • Stratified sampling preserves distributions
   • Incremental learning for very large datasets

Common Challenges

Curse of Dimensionality

Problem: Performance degrades with many features Solutions:

• Feature selection
• Dimensionality reduction (PCA, t-SNE)
• Regularization

Imbalanced Data

Problem: Rare class underrepresented Solutions:

• Resampling (SMOTE, undersampling)
• Class weights
• Anomaly detection instead of classification

Scalability

Problem: Algorithms don't scale to large datasets Solutions:

• Mini-batch processing
• Approximate algorithms
• Distributed computing (Spark MLlib)

Using the Reference Files

When to Read Each Reference

/references/association-rules-advanced.md — Read when implementing market basket analysis, optimizing Apriori/FP-Growth parameters, or working with large transaction datasets.

/references/clustering-comprehensive.md — Read when performing customer segmentation, comparing clustering algorithms, or evaluating cluster quality.

/references/classification-methods.md — Read when building predictive models, handling imbalanced data, or optimizing classification performance.

/references/anomaly-detection-techniques.md — Read when detecting fraud, identifying network intrusions, or finding unusual patterns in data.

/references/data-mining-case-studies.md — Read when starting a new data mining project or looking for real-world application examples.