Outlier Detective

Outlier Detective

Detect anomalies and outliers in numeric data using multiple methods.

Features

• Statistical Methods: Z-score, IQR, Modified Z-score
• ML Methods: Isolation Forest, LOF, DBSCAN
• Visualization: Box plots, scatter plots
• Multi-Column: Analyze multiple variables
• Reports: Detailed outlier reports
• Flexible Thresholds: Configurable sensitivity

Quick Start

from outlier_detective import OutlierDetective

detective = OutlierDetective()
detective.load_csv("sales_data.csv")

# Detect outliers in a column
outliers = detective.detect("revenue", method="iqr")
print(f"Found {len(outliers)} outliers")

# Get full report
report = detective.analyze("revenue")
print(report)

CLI Usage

# Detect outliers using IQR method
python outlier_detective.py --input data.csv --column sales --method iqr

# Use Z-score with custom threshold
python outlier_detective.py --input data.csv --column price --method zscore --threshold 3

# Analyze all numeric columns
python outlier_detective.py --input data.csv --all

# Generate visualization
python outlier_detective.py --input data.csv --column revenue --plot boxplot.png

# Export outliers to CSV
python outlier_detective.py --input data.csv --column value --output outliers.csv

# Use Isolation Forest (ML)
python outlier_detective.py --input data.csv --method isolation_forest

API Reference

OutlierDetective Class

class OutlierDetective:
    def __init__(self)

    # Data loading
    def load_csv(self, filepath: str, **kwargs) -> 'OutlierDetective'
    def load_dataframe(self, df: pd.DataFrame) -> 'OutlierDetective'

    # Detection (single column)
    def detect(self, column: str, method: str = "iqr", **kwargs) -> pd.DataFrame
    def analyze(self, column: str) -> dict

    # Detection (multi-column)
    def detect_multivariate(self, columns: list = None, method: str = "isolation_forest") -> pd.DataFrame
    def analyze_all(self) -> dict

    # Visualization
    def plot_boxplot(self, column: str, output: str) -> str
    def plot_scatter(self, col1: str, col2: str, output: str) -> str
    def plot_distribution(self, column: str, output: str) -> str

    # Export
    def get_outliers(self, column: str, method: str = "iqr") -> pd.DataFrame
    def get_clean_data(self, column: str, method: str = "iqr") -> pd.DataFrame

Detection Methods

Statistical Methods

IQR (Interquartile Range)

• Default and most robust method
• Outliers: values below Q1 - 1.5×IQR or above Q3 + 1.5×IQR
• Multiplier configurable (default: 1.5)

outliers = detective.detect("price", method="iqr", multiplier=1.5)

Z-Score

• Based on standard deviations from mean
• Assumes normal distribution
• Threshold configurable (default: 3)

outliers = detective.detect("price", method="zscore", threshold=3)

Modified Z-Score

• Uses median instead of mean
• More robust to existing outliers
• Based on MAD (Median Absolute Deviation)

outliers = detective.detect("price", method="modified_zscore", threshold=3.5)

ML Methods

Isolation Forest

• Ensemble method, good for high-dimensional data
• Contamination parameter sets expected outlier fraction

outliers = detective.detect_multivariate(
    method="isolation_forest",
    contamination=0.1
)

Local Outlier Factor (LOF)

• Density-based method
• Compares local density to neighbors

outliers = detective.detect_multivariate(
    method="lof",
    n_neighbors=20
)

Output Format

detect() Result

# Returns DataFrame of outlier rows with additional columns:
#   - outlier_score: How extreme the value is
#   - outlier_reason: Description of why it's an outlier

   index  value  outlier_score  outlier_reason
0     15   5000          4.2    Above Q3 + 1.5×IQR
1     42  -1000         -3.8    Below Q1 - 1.5×IQR

analyze() Result

{
    "column": "revenue",
    "total_rows": 1000,
    "outlier_count": 23,
    "outlier_percent": 2.3,
    "methods": {
        "iqr": {"count": 23, "indices": [...]},
        "zscore": {"count": 18, "indices": [...]},
        "modified_zscore": {"count": 20, "indices": [...]}
    },
    "stats": {
        "mean": 5432.10,
        "median": 4890.00,
        "std": 1234.56,
        "min": -1000.00,
        "max": 15000.00,
        "q1": 3500.00,
        "q3": 6200.00,
        "iqr": 2700.00
    },
    "bounds": {
        "lower": -550.00,
        "upper": 10250.00
    }
}

Example Workflows

Data Cleaning Pipeline

detective = OutlierDetective()
detective.load_csv("raw_data.csv")

# Analyze and visualize
report = detective.analyze("price")
print(f"Found {report['outlier_count']} outliers ({report['outlier_percent']:.1f}%)")

# Get clean data
clean_data = detective.get_clean_data("price", method="iqr")
clean_data.to_csv("clean_data.csv")

Fraud Detection

detective = OutlierDetective()
detective.load_csv("transactions.csv")

# Use multiple methods for consensus
iqr_outliers = set(detective.detect("amount", method="iqr").index)
zscore_outliers = set(detective.detect("amount", method="zscore").index)

# Transactions flagged by both methods
high_confidence = iqr_outliers & zscore_outliers
print(f"High-confidence anomalies: {len(high_confidence)}")

Multi-Variable Analysis

detective = OutlierDetective()
detective.load_csv("sensors.csv")

# Detect multivariate outliers
outliers = detective.detect_multivariate(
    columns=["temp", "pressure", "humidity"],
    method="isolation_forest",
    contamination=0.05
)
print(f"Anomalous readings: {len(outliers)}")

Visualization Examples

# Box plot with outliers highlighted
detective.plot_boxplot("revenue", "revenue_boxplot.png")

# Distribution with bounds
detective.plot_distribution("price", "price_dist.png")

# Scatter plot (2D outliers)
detective.plot_scatter("feature1", "feature2", "scatter.png")

Dependencies

• pandas>=2.0.0
• numpy>=1.24.0
• scipy>=1.10.0
• scikit-learn>=1.3.0
• matplotlib>=3.7.0