How It Works

Meta-Analyzer

The Meta-Analyzer is an optional second-pass LLM analysis feature that reviews findings from other analyzers to improve accuracy and provide actionable insights. In our testing, it significantly reduces false positives while preserving threat detection capability.

Overview

When enabled via the --enable-meta CLI flag or enable_meta API parameter, the Meta-Analyzer performs:

• False Positive Filtering: Identifies genuinely benign findings based on full skill context (only marks FPs when code is actually safe)
• Finding Correlation: Groups related findings from different analyzers into logical threat groups (e.g., 4 autonomy-abuse YARA matches = 1 correlation group)
• Priority Ranking: Ranks findings by actual exploitability and business impact
• Remediation Guidance: Provides specific, actionable recommendations per correlation group
• Risk Assessment: Overall skill verdict (SAFE/SUSPICIOUS/MALICIOUS) with reasoning
• Confidence Enrichment: Adds meta_confidence, meta_exploitability, and meta_impact to every validated finding

The meta-analyzer has full access to skill content (SKILL.md and code files), which helps it validate whether pattern-based detections are likely real threats.

How It Works

1. Collect Findings: All selected analyzers (static, behavioral, LLM, etc.) run and produce findings
2. Aggregate Full Context: The meta-analyzer receives:
   • All findings from other analyzers
   • Complete SKILL.md content (up to 50KB)
   • All code files (.py, .js, .ts, .sh, .bash) with content (up to 30KB per file, 150KB total)
3. Authority-Based Review: Uses an analyzer authority hierarchy to weight findings:
   • LLM Analyzer (highest) > Behavioral > AI Defense > Static/Pipeline/Bytecode > Trigger (lowest)
4. Validate & Correlate:
   • Each finding is verified against actual code content
   • Genuinely benign findings are marked as false positives
   • Related findings are grouped into correlation groups
   • A follow-up pass covers any findings the initial analysis missed
5. Enrich Findings: Each validated finding receives:
   • meta_confidence: HIGH/MEDIUM/LOW with reasoning
   • meta_exploitability: How easy it is to exploit
   • meta_impact: Business/security impact assessment
   • Specific remediation recommendations

CLI Usage

# Scan with static + LLM + meta-analysis
skill-scanner scan /path/to/skill --use-llm --enable-meta

# Scan with behavioral + LLM + meta-analysis (recommended for best results)
skill-scanner scan /path/to/skill --use-behavioral --use-llm --enable-meta

# Scan all skills in a directory
skill-scanner scan-all /path/to/skills --use-llm --enable-meta --recursive

# Output in JSON format
skill-scanner scan /path/to/skill --use-llm --enable-meta --format json

Requirements:

• Minimum 2 analyzers: Meta-analysis requires at least 2 analyzers (e.g., static + LLM, or behavioral + LLM). This ensures the meta-analyzer has multiple perspectives to correlate.
• LLM API key: An API key must be configured for the meta-analyzer's LLM calls (see Configuration section for supported providers)
• Recommended: Use --use-llm with meta-analysis for best results, as LLM findings provide the semantic understanding the meta-analyzer relies on

API Usage

All scan endpoints support the enable_meta parameter:

curl -X POST "http://localhost:8000/scan" \
  -H "Content-Type: application/json" \
  -d '{
    "skill_directory": "/path/to/skill",
    "use_llm": true,
    "use_behavioral": true,
    "enable_meta": true
  }'

Configuration

The meta-analyzer can use a separate LLM API key from the primary LLM analyzer. This allows you to:

• Use a different model for meta-analysis (e.g., GPT-4 for meta, Claude for primary)
• Use separate rate limits/quotas
• Route through different endpoints

Environment Variables

The scanner uses SKILL_SCANNER_* environment variables exclusively (no provider-specific fallbacks to avoid accidentally using other keys).

Scanner-wide settings (apply to both LLM and Meta analyzers):

export SKILL_SCANNER_LLM_API_KEY="your-api-key"
export SKILL_SCANNER_LLM_MODEL="claude-3-5-sonnet-20241022"
export SKILL_SCANNER_LLM_BASE_URL="https://..."  # For Azure/custom endpoints
export SKILL_SCANNER_LLM_API_VERSION="2025-01-01-preview"  # For Azure

Meta-specific overrides (optional - use different model/key for meta-analysis):

export SKILL_SCANNER_META_LLM_API_KEY="different-key"
export SKILL_SCANNER_META_LLM_MODEL="gpt-4o"
export SKILL_SCANNER_META_LLM_BASE_URL="https://..."
export SKILL_SCANNER_META_LLM_API_VERSION="..."

Priority Order

Special Cases

Setup Examples

# Standard setup (one key for everything)
export SKILL_SCANNER_LLM_API_KEY="sk-ant-..."
export SKILL_SCANNER_LLM_MODEL="claude-3-5-sonnet-20241022"

# Azure OpenAI setup
export SKILL_SCANNER_LLM_API_KEY="your-azure-key"
export SKILL_SCANNER_LLM_MODEL="azure/gpt-4.1"
export SKILL_SCANNER_LLM_BASE_URL="https://your-resource.openai.azure.com/"
export SKILL_SCANNER_LLM_API_VERSION="2025-01-01-preview"

# Separate meta key for second opinion (advanced)
export SKILL_SCANNER_LLM_API_KEY="sk-ant-..."  # Primary: Claude
export SKILL_SCANNER_META_LLM_API_KEY="sk-..."  # Meta: OpenAI
export SKILL_SCANNER_META_LLM_MODEL="gpt-4o"

Provider Examples

Anthropic Claude:

export SKILL_SCANNER_LLM_API_KEY="sk-ant-..."
export SKILL_SCANNER_LLM_MODEL="claude-3-5-sonnet-20241022"

OpenAI:

export SKILL_SCANNER_LLM_API_KEY="sk-..."
export SKILL_SCANNER_LLM_MODEL="gpt-4o"

Azure OpenAI:

export SKILL_SCANNER_LLM_API_KEY="your-azure-key"
export SKILL_SCANNER_LLM_MODEL="azure/gpt-4.1"
export SKILL_SCANNER_LLM_BASE_URL="https://your-resource.openai.azure.com/"
export SKILL_SCANNER_LLM_API_VERSION="2025-01-01-preview"

Google Gemini:

export SKILL_SCANNER_LLM_API_KEY="your-gemini-key"
export SKILL_SCANNER_LLM_MODEL="gemini/gemini-1.5-pro"

AWS Bedrock:

export SKILL_SCANNER_LLM_API_KEY="bedrock-api-key-..."
export SKILL_SCANNER_LLM_MODEL="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"

Google Vertex AI (uses service account):

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
export SKILL_SCANNER_LLM_MODEL="vertex_ai/gemini-1.5-pro"

Analyzer Authority Hierarchy

The meta-analyzer weights findings based on which analyzer produced them:

Authority-Based Rules

• LLM + Behavioral agree → HIGH confidence true positive
• LLM says SAFE, Static flags pattern-only (no malicious context) → Likely false positive
• LLM says THREAT, others missed → True positive (trust LLM)
• Only Static flagged, but code confirms the issue → True positive (MEDIUM confidence)
• Only Static flagged, keyword-only with no malicious context → Likely false positive
• Multiple analyzers flag different aspects of same issue → Correlated — group, keep all

Output Format

Meta-analyzed findings include enriched metadata:

{
  "id": "meta_finding_1",
  "rule_id": "META_VALIDATED",
  "category": "data_exfiltration",
  "severity": "HIGH",
  "title": "Credential Theft via Network Exfiltration",
  "description": "Skill reads AWS credentials and sends to external server",
  "file_path": "scripts/sync.py",
  "line_number": 42,
  "remediation": "Remove network call or use secure credential management",
  "analyzer": "meta",
  "metadata": {
    "meta_validated": true,
    "meta_confidence": "HIGH",
    "meta_confidence_reason": "Clear source→sink flow from credential file to external POST",
    "meta_exploitability": "Easy - no authentication required",
    "meta_impact": "Critical - AWS credential compromise",
    "aitech": "AITech-8.2"
  }
}

Finding Correlation

When multiple analyzers report overlapping threats, the meta-analyzer groups them into correlation groups rather than removing them. All findings are preserved in validated_findings, and the correlations block shows how they relate. For example:

{
  "correlations": [
    {
      "group_name": "Credential Theft Chain",
      "finding_indices": [3, 12, 13, 24, 46],
      "relationship": "Pipeline taint flows and static pattern matches all confirm credential exfiltration",
      "combined_severity": "CRITICAL",
      "consolidated_remediation": "Remove all credential exfiltration code and network calls to untrusted endpoints"
    }
  ]
}

This preserves the granular evidence from each analyzer (line numbers, exact patterns, taint chains) while providing the consolidated view for executive reporting. Use --verbose to also include findings the meta-analyzer marked as false positives.

Visual Reports

Correlation groups can be visualized as Mermaid attack-path diagrams in two formats:

• Markdown (--format markdown): Each correlation group renders as a mermaid code block showing source files → detections → attack outcome, followed by a findings table and remediation guidance.
• HTML (--format html): A self-contained, interactive HTML report with:
  • Risk verdict banner and severity bar
  • Collapsible correlation group cards with embedded Mermaid diagrams
  • Prioritized recommendation cards with effort badges
  • Sortable findings table

# Generate an interactive HTML report
skill-scanner scan /path/to/skill --use-llm --enable-meta --format html --output report.html

AITech Taxonomy

The meta-analyzer uses the AITech taxonomy for threat classification:

Evaluating Meta-Analyzer Performance

The eval runner supports comparing results with and without the meta-analyzer:

# Run comparison evaluation
uv run python evals/runners/eval_runner.py --compare

# With detailed per-skill breakdown
uv run python evals/runners/eval_runner.py --compare --show-details

Sample Comparison Output

======================================================================
COMPARISON RESULTS
======================================================================

Metric                       Without Meta       With Meta       Change
----------------------------------------------------------------------
True Positives                         40              16          -24
False Positives                        17              11           -6
----------------------------------------------------------------------
Meta-Analyzer Impact:
  Total findings filtered: 50
  Total findings validated: 27
  Noise reduction rate: 64.9%

======================================================================
SUMMARY
======================================================================

Safe Skills Detection:   2/2 -> 2/2
Unsafe Skills Detection: 9/9 -> 9/9

Summary:
  Meta-Analyzer filtered 50 low-value findings
  while maintaining 9/9 unsafe skill detection rate

  Meta-analyzer improved signal-to-noise without reducing unsafe-skill detection in this sample run.

The comparison shows that while raw metrics like "true positives" decrease (because redundant pattern matches are consolidated), the threat detection capability is preserved. Results may vary depending on your specific skills and threat patterns.

Best Practices

1. Use multiple analyzers: Meta-analysis is most effective when correlating findings from 2+ analyzers
2. Include LLM analyzer: The LLM analyzer provides the semantic understanding meta-analysis relies on
3. Review filtered findings: Check verbose output for false positives that were filtered
4. Configure appropriate model: Use a capable model (GPT-4, Claude 3.5+) for best results
5. Consider latency: Meta-analysis adds one LLM call, increasing scan time by ~5-15 seconds
6. Use --compare for validation: Run the eval comparison to verify meta-analyzer effectiveness on your skills

Troubleshooting

Meta-analyzer not running:

• Ensure --enable-meta flag is provided
• Verify at least 2 analyzers are enabled
• Check that LLM API key is configured
• Look for "Using Meta-Analyzer" in output

No findings after meta-analysis:

• All findings may have been filtered as false positives
• Check verbose output: "X false positives filtered"
• This can be correct behavior for benign skills

Slow scans with meta-analysis:

• Meta-analysis adds one LLM API call per skill
• Consider using a faster model via SKILL_SCANNER_META_LLM_MODEL
• For batch scans, meta-analysis runs per-skill (not aggregated)

Different results than expected:

• Meta-analyzer uses authority hierarchy - LLM findings take precedence
• Pattern-only matches from static analyzer may be filtered as FPs
• Check meta_confidence_reason for explanation