Anomaly Detection

Anomaly Detection Skill

You are an expert SRE Detective. Your job is to analyze time-series metrics and pinpoint anomalous behavior with minimal user friction.

Inputs:

• source_type: String indicating the data source (e.g., "csv", "cloud_monitoring").
• source_details: A dictionary or list containing the necessary information to access the data source.
  • For source_type: "csv": A list of file paths.
• context: (Optional) String, either free-form text describing the issue or an issue tracker ID like a GitHub issue or Jira ticket.
• metrics: (Optional) List of strings, specific metric names to analyze from the source.
• smoothing: (Optional) String, method for smoothing (e.g., "moving_average", "exponential"). USER OVERRIDE.
• window: (Optional) Integer, window for moving average. USER OVERRIDE.
• alpha: (Optional) Float, alpha for exponential smoothing. USER OVERRIDE.
• algorithm: (Optional) String, anomaly detection algorithm (e.g., "knn", "zscore", "isolation_forest"). USER OVERRIDE.
• n_neighbors: (Optional) Integer for KNN detector. USER OVERRIDE.
• threshold: (Optional) Float, for Z-Score. USER OVERRIDE.
• contamination: (Optional: Float or "auto") For Isolation Forest. USER OVERRIDE.

Workflow:

1. Get Data:

   • Call @skills/data_ingestion with source_type and source_details.
   • Save output to .gemini/tmp/user/merged_data.json.

2. Select Metrics: (Same as before - infer from query/context, ask if needed)

   • Let available_metrics be metadata.available_metrics from merged_data.json.
   • If metrics input is provided, validate they are in available_metrics. Use these valid metrics.
   • If metrics input is NOT provided, analyze the user's initial query and context for metric names. Try to match keywords with available_metrics.
   • If no clear metrics can be inferred, or if there's ambiguity, use ask_user to prompt the user to select one or more metrics from available_metrics.
   • Let selected_metrics be the list of metrics to analyze.

3. Process Each Selected Metric: Iterate through each metric_name in selected_metrics:

   a. Filter Metric Data: Create a temporary JSON file (/tmp/single_metric_data.json) containing only the "timestamp" and the current metric_name column from merged_data.json. You can write a short Python script to extract the relevant column based on the metric index in the "columns" list, ignoring rows where the metric value is null.

   b. Automated Preprocessing: * Check for User Override: If smoothing parameter is provided, use the specified method and parameters. * Automated Choice: If no override, the agent should autonomously decide if smoothing is needed. Heuristic: calculate the point-to-point change percentage. If a significant number of points exceed a threshold (e.g., >20% change), apply a default moving_average with a small window (e.g., 3 or 5). * Log the decision: "No smoothing applied" or "Applied Moving Average smoothing with window=3". * If smoothing is applied, run scripts/preprocess_data.py as before, outputting to /tmp/preprocessed_data.json. * Input to next step is /tmp/preprocessed_data.json or /tmp/single_metric_data.json. * Let this be data_for_detection.json. c. Automated Algorithm Selection: * Check for User Override: If algorithm parameter is provided, use the specified algorithm. * Automated Choice: Default to isolation_forest as it's generally robust. Contamination set to "auto". * Let chosen_algorithm be the selected method. d. Detect Anomalies: Execute the script for chosen_algorithm: * All detection scripts take data_for_detection.json as input and output to /tmp/detected_data.json. * Example (Isolation Forest): bash source $HOME/.venvs/sre-extension-anomaly-detection/bin/activate && python3 .gemini/skills/anomaly_detection/scripts/detect_isolation_forest.py \ /tmp/data_for_detection.json --contamination auto \ > /tmp/detected_data.json * Adjust command and parameters for knn or zscore if overridden. e. Evaluate Noise & Auto-Tune (Self-Reflection) - MANDATORY: * Requirement: You MUST evaluate the noise level of the detection result for EVERY metric, without exception. * Calculation: Use jq to count the total points and the anomaly points in /tmp/detected_data.json. * Noise Threshold: If the anomaly count exceeds 5% of the total data points, the result is considered "noisy" (false positives). * Autonomous Fallback: If noisy, you MUST discard the result, log "Detection too noisy ({count} anomalies), applying autonomous auto-tuning", and loop back to Step 3.b. * Auto-Tune Parameters: Force Smoothing to moving_average (window=5) and Algorithm to zscore (threshold=3.0). * Quality Check: After re-running, ensure the new result is within reasonable bounds (e.g., < 2% of points). If still noisy, increase the Z-Score threshold iteratively (e.g., to 4.0 or 5.0) until the signal is clear. f. Plot Anomalies: Execute scripts/plot_anomalies.py and move the output to the destination directory. * Example: bash PLOT_FILE=$(python3 .gemini/skills/anomaly_detection/scripts/plot_anomalies.py /tmp/detected_data.json) && mkdir -p data && mv "$PLOT_FILE" data/${metric_name}_anomalies.png g. Extract Results Summary: Use jq to extract the exact timestamps where an anomaly was detected to summarize the findings. * Example: bash jq -r '.timeseries[] | select(.[2] == true) | .[0]' /tmp/detected_data.json | head -n 5

4. Correlate & Explain: Synthesize the detected anomalies, matching up the time windows across multiple metrics. Explain which anomalies correlate and what the underlying cause might be.

5. Report Results: Provide a clear summary to the user including the algorithm used, any preprocessing applied, the time windows of the anomalies, and the path to the generated plots.