Evaluation Framework Guide

Evaluation Framework Guide

How Evaluation Works

Overview

The evaluation framework (evals/eval_runner.py) tests analyzer accuracy by comparing actual scan results against expected ground truth defined in _expected.json files.

Evaluation Process

1. Load Expected Results: Reads _expected.json from each skill directory
2. Run Scanners: Scans skill with configured analyzers (Static, LLM, Behavioral)
3. Compare Results: Matches actual findings against expected findings
4. Calculate Metrics: Computes precision, recall, F1 score, etc.

Analyzer Configuration

The evaluation can run with different analyzer combinations:

Static Analyzer Only (Default)

python evals/eval_runner.py --test-skills-dir evals/skills

• Uses StaticAnalyzer (includes YARA rules, pattern matching)
• Fast, no API calls needed
• Good for testing YARA rules and static patterns

Static + LLM Analyzer

export SKILL_SCANNER_LLM_API_KEY=your_key
python evals/eval_runner.py --test-skills-dir evals/skills --use-llm

• Uses StaticAnalyzer + LLMAnalyzer
• Combines pattern matching with semantic analysis
• More comprehensive but slower (API calls)

With Meta-Analyzer (False Positive Filtering)

python evals/eval_runner.py --test-skills-dir evals/skills --use-llm --use-meta

• Adds MetaAnalyzer for second-pass analysis
• Filters false positives, consolidates redundant findings
• Best for production-quality results with reduced noise

Compare With/Without Meta-Analyzer

python evals/eval_runner.py --test-skills-dir evals/skills --use-llm --compare

• Runs evaluation twice: with and without meta-analysis
• Shows per-skill changes and overall impact
• Validates meta-analyzer is reducing noise without missing threats

Static + Behavioral Analyzer

# Would need to add --use-behavioral flag
# Uses dataflow analysis for cross-file threats

Expected Results Format

Each skill directory should have an _expected.json file:

{
  "skill_name": "skill-name",
  "expected_safe": false,
  "expected_severity": "CRITICAL",
  "expected_findings": [
    {
      "category": "prompt_injection",
      "severity": "HIGH",
      "description": "Contains instruction override attempt"
    },
    {
      "category": "data_exfiltration",
      "severity": "CRITICAL",
      "description": "Sends data to external server"
    }
  ],
  "notes": "Additional context about the skill"
}

Note: The framework supports both expected_findings and expected_threats keys for backward compatibility.

Matching Logic

The evaluation matches findings based on:

1. Category Match: Finding category must match expected category
2. Severity Match: Finding severity must match expected severity
3. One-to-One Matching: Each expected finding matches at most one actual finding

Example:

• Expected: {category: "prompt_injection", severity: "HIGH"}
• Actual: {category: "prompt_injection", severity: "HIGH"} [OK] Match
• Actual: {category: "prompt_injection", severity: "CRITICAL"} [NO MATCH] No match (severity mismatch)

Metrics Calculation

For Safe Skills (expected_safe: true)

• True Negative (TN): No findings detected [OK]
• False Positive (FP): Any finding detected [NO MATCH]
• Correct: actual_safe == true && false_positives == 0

For Unsafe Skills (expected_safe: false)

• True Positive (TP): Expected finding matched [OK]
• False Negative (FN): Expected finding not found [NO MATCH]
• False Positive (FP): Unmatched finding (only if we're also missing expected findings)
• Correct: actual_safe == false && false_negatives == 0

Important: Extra findings beyond expected ones are NOT counted as false positives if all expected findings are found. This is because finding MORE threats is actually GOOD!

Precision and Recall

• Precision = TP / (TP + FP)

  • Measures: "Of all findings, how many were correct?"
  • High precision = Low false positive rate

• Recall = TP / (TP + FN)

  • Measures: "Of all expected threats, how many did we find?"
  • High recall = Low false negative rate

• F1 Score = 2 × (Precision × Recall) / (Precision + Recall)

  • Harmonic mean of precision and recall
  • Balanced metric

Current Issues and Solutions

Issue 1: Low Precision/Recall

Problem: Expected findings might be incomplete - LLM analyzer finds more threats than documented.

Solution: Update _expected.json files to include all legitimate threats found by analyzers.

Example: If LLM finds 5 threats but _expected.json only lists 2, update it to include all 5.

Issue 2: Severity Mismatches

Problem: Expected severity might be "HIGH" but analyzer reports "CRITICAL".

Solution: Either:

• Update expected severity to match analyzer output
• Or make matching more flexible (category-only matching)

Issue 3: Category Variations

Problem: Analyzer uses different category names than expected.

Solution: The framework already has category mapping (e.g., sql_injection → command_injection).

Improving Precision/Recall

1. Update Expected Findings

Review actual findings and update _expected.json:

# Scan a skill and see what it finds
python -m skill_scanner.cli.cli scan evals/skills/prompt-injection/jailbreak-override --use-llm --format json > actual_findings.json

# Compare with expected and update _expected.json

2. Use Flexible Matching

Consider matching by category only (ignore severity) for some evaluations:

# In eval_runner.py, could add:
match_by_category_only = True  # Option to match by category only

3. Separate Analyzer Evaluation

Evaluate each analyzer separately to understand their strengths:

# Test Static only
python evals/eval_runner.py --test-skills-dir evals/skills

# Test LLM only (would need modification)
# Test Static + LLM
python evals/eval_runner.py --test-skills-dir evals/skills --use-llm

4. Expected Findings Best Practices

• Be Comprehensive: Include ALL legitimate threats, not just primary ones
• Match Analyzer Output: If analyzer finds CRITICAL, expect CRITICAL
• Use Correct Categories: Match the ThreatCategory enum values
• Document Patterns: Use threat_patterns array to document attack patterns

Example: Updating Expected Findings

Before (jailbreak-override/_expected.json):

{
  "expected_threats": [
    {"category": "prompt_injection", "severity": "HIGH"}
  ]
}

After (based on actual findings):

{
  "expected_findings": [
    {"category": "prompt_injection", "severity": "HIGH"},
    {"category": "prompt_injection", "severity": "CRITICAL"},
    {"category": "policy_violation", "severity": "HIGH"}
  ]
}

Running Evaluations

# Set environment variables
export SKILL_SCANNER_LLM_API_KEY=your_key
export SKILL_SCANNER_LLM_MODEL=claude-3-5-sonnet-20241022

# Basic evaluation (Static analyzer only)
python evals/eval_runner.py --test-skills-dir evals/skills

# With LLM analyzer
python evals/eval_runner.py --test-skills-dir evals/skills --use-llm

# With Meta-Analyzer (filters false positives)
python evals/eval_runner.py --test-skills-dir evals/skills --use-llm --use-meta

# Compare with and without meta-analyzer
python evals/eval_runner.py --test-skills-dir evals/skills --use-llm --compare

# Save results
python evals/eval_runner.py --test-skills-dir evals/skills --use-llm --output results.json

Understanding Results

Metrics:
  accuracy: 90.91%      # Overall correctness
  precision: 66.67%     # TP / (TP + FP) - Low false positives
  recall: 66.67%        # TP / (TP + FN) - Low false negatives
  f1_score: 66.67%      # Balanced metric
  true_positives: 2     # Correctly found threats
  false_positives: 1    # Incorrect findings
  true_negatives: 2     # Correctly identified safe skills
  false_negatives: 1    # Missed threats

Recommendations

1. Update Expected Findings: Review actual LLM findings and update _expected.json files to be comprehensive
2. Test Separately: Evaluate Static vs LLM vs Combined to understand each analyzer's contribution
3. Use Meta-Analyzer: Run with --use-meta for production-quality results with fewer false positives
4. Compare Impact: Use --compare to validate meta-analyzer is effective for your skill set
5. Category Flexibility: Consider matching by category only for some evaluations
6. Document Patterns: Use threat_patterns to document attack patterns even if not matching on them