Vibe Pentest AI Security Testing

vibe-pentest-ai-security-testing

Skill by ara.so — Security Skills collection.

Vibe Pentest is an AI Agent-based automated penetration testing tool that uses a multi-agent parallel execution architecture to perform comprehensive black-box penetration testing (including business logic vulnerability assessment) on web applications, APIs, and admin backends. It outputs stable and reliable security reports with actionable remediation recommendations.

Overview

Vibe Pentest orchestrates multiple AI agents to:

• Fingerprint web technologies and frameworks
• Crawl and map attack surfaces
• Execute parallel vulnerability testing across multiple categories
• Identify business logic flaws
• Generate comprehensive security reports in HTML and DOCX formats

Version: v1.0.7
License: AGPL-3.0
Primary Language: Python

Installation

Prerequisites

1. Git (required for auto-update mechanism):

# Clone from Gitee (recommended for better access)
git clone https://gitee.com/ok-helloworld/vibe-pentest
cd vibe-pentest

# Or from GitHub
git clone https://github.com/ok-helloworld/vibe-pentest
cd vibe-pentest

2. Python 3.10+ and dependencies:

pip install playwright python-docx matplotlib requests urllib3 argparse httpx charset-normalizer chardet
playwright install chromium

3. Katana Crawler (included for Windows, download for other OS):
   • Windows version included in tools/katana
   • For other OS, see tools/katana_downloads.json for download links

Automated Installation

You can also ask your AI coding agent to install everything:

Install vibe-pentest skill including all runtime dependencies from: https://gitee.com/ok-helloworld/vibe-pentest

Project Structure

vibe-pentest/
├── scripts/              # Core testing scripts
│   ├── run_katana.py    # Crawler wrapper
│   ├── prepare_agent_findings.py  # Multi-agent orchestration
│   ├── generate_report.py         # Report generation
│   └── ...
├── tools/               # External tools (katana, etc.)
├── workspace/           # Test outputs (created during execution)
│   ├── sessions/        # Browser session data
│   ├── findings/        # Vulnerability findings
│   └── report_result/   # Final reports
└── prompts/             # AI prompt templates

Core Testing Workflow

Vibe Pentest follows a 7-phase workflow:

Phase 0: Fingerprinting

Identify web technologies, frameworks, and server information.

# Example: Running fingerprint detection
import subprocess
import json

result = subprocess.run(
    ["python", "scripts/fingerprint.py", "--url", "https://example.com"],
    capture_output=True,
    text=True
)

fingerprint = json.loads(result.stdout)
print(f"Detected: {fingerprint.get('framework')}, {fingerprint.get('server')}")

Phase 0.5: Backend Entry Scanning

Scan for admin panels and sensitive endpoints.

Phase 1: Authorization Confirmation

Verify written authorization before proceeding.

Phase 2: Browser Login & Credential Extraction

Launch browser for manual login, extract session cookies/tokens.

# Example: Browser session extraction
from playwright.sync_api import sync_playwright

def extract_session(target_url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)
        context = browser.new_context()
        page = context.new_page()
        
        page.goto(target_url)
        input("Press Enter after logging in manually...")
        
        # Extract cookies and local storage
        cookies = context.cookies()
        storage = page.evaluate("() => Object.assign({}, localStorage)")
        
        browser.close()
        return {"cookies": cookies, "storage": storage}

Phase 3: Katana Crawling

Use Katana crawler to discover all endpoints and parameters.

# Example: Running Katana crawler via script
import subprocess
import time

# Run crawler (must execute outside sandbox)
proc = subprocess.Popen(
    ["python", "scripts/run_katana.py", 
     "--url", "https://example.com",
     "--cookies", "session=abc123"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

# Monitor for max 20 minutes
timeout = 1200
start_time = time.time()
while proc.poll() is None and (time.time() - start_time) < timeout:
    time.sleep(10)

if proc.poll() is None:
    proc.terminate()
    time.sleep(5)  # Wait for results to flush

# Read crawl results
with open("workspace/crawl_summary.json") as f:
    crawl_data = json.load(f)

Phase 4: Data Cleaning

Process crawler output, deduplicate URLs, extract parameters.

Phase 4.5: Attack Surface Mapping

Map discovered endpoints to vulnerability test categories.

Phase 5: Multi-Agent Parallel Testing

Distribute testing across 6 specialized agents using prepared skeleton files.

# Example: Preparing agent findings skeleton
import subprocess
import json

# Generate skeleton findings for 6 agents
subprocess.run([
    "python", "scripts/prepare_agent_findings.py",
    "--targets", "workspace/targets.txt",
    "--fingerprint", "workspace/fingerprint.json",
    "--output", "workspace/findings"
])

# Each agent gets assigned specific test categories:
# Agent 1: Authentication & Authorization
# Agent 2: Injection Attacks (SQLi, XSS, etc.)
# Agent 3: Business Logic & IDOR
# Agent 4: File Upload & Path Traversal
# Agent 5: API Security & Rate Limiting
# Agent 6: Information Disclosure & Misconfigurations

Phase 5.5: Attack Chain Analysis

Identify cross-agent attack chains and compound vulnerabilities.

Phase 5.6: Evidence Verification

Re-verify confirmed vulnerabilities with HTTP evidence.

# Example: Verifying SQLi finding
import httpx

def verify_sqli(endpoint, payload, original_response_time):
    # Time-based SQLi verification
    url = f"{endpoint}?id={payload}"
    
    start = time.time()
    response = httpx.get(url, timeout=30)
    elapsed = time.time() - start
    
    if elapsed > original_response_time + 5:
        return {
            "verified": True,
            "method": "GET",
            "url": url,
            "response_time": elapsed,
            "status_code": response.status_code
        }
    return {"verified": False}

Phase 6: Report Generation

Generate comprehensive reports in multiple formats.

# Example: Generating final report
import subprocess
from datetime import datetime

timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")

subprocess.run([
    "python", "scripts/generate_report.py",
    "--findings-dir", "workspace/findings",
    "--fingerprint", "workspace/fingerprint.json",
    "--output-json", f"workspace/report_{timestamp}.json",
    "--output-html", f"workspace/report_{timestamp}.html",
    "--output-docx", f"workspace/report_{timestamp}.docx"
])

Configuration

Environment Variables

# Set custom workspace directory
export VIBE_WORKSPACE="/path/to/workspace"

# Configure crawler timeout (seconds)
export KATANA_TIMEOUT=1200

# Set max concurrent agents
export MAX_AGENTS=6

# Configure LLM provider (for AI agents)
export OPENAI_API_KEY=your_key_here
export ANTHROPIC_API_KEY=your_key_here

Testing Principles

Critical Rules:

1. All sub-agents MUST actively investigate, not wait for prompts
2. If Katana runs >20 minutes, terminate and collect results
3. Test ALL discovered functionality, not just entry points
4. Attempt 2-3 bypass techniques on failed tests
5. Iron Law: May modify/delete own test data; NEVER modify production data

Common Usage Patterns

Standard Authorized Testing

"""
Complete penetration test workflow with single account
"""
import os
import subprocess
import json

def run_standard_pentest(target_url, auth_statement, account_info):
    workspace = "workspace"
    os.makedirs(workspace, exist_ok=True)
    
    # Phase 0: Fingerprinting
    print("[Phase 0] Fingerprinting...")
    subprocess.run(["python", "scripts/fingerprint.py", 
                   "--url", target_url,
                   "--output", f"{workspace}/fingerprint.json"])
    
    # Phase 0.5: Backend scanning
    print("[Phase 0.5] Scanning for admin panels...")
    subprocess.run(["python", "scripts/admin_scanner.py",
                   "--url", target_url,
                   "--output", f"{workspace}/admin_entries.json"])
    
    # Phase 1: Confirm authorization
    print(f"[Phase 1] Authorization: {auth_statement}")
    
    # Phase 2: Extract session
    print("[Phase 2] Launch browser for manual login...")
    session_data = extract_session(target_url)
    with open(f"{workspace}/sessions/session.json", "w") as f:
        json.dump(session_data, f)
    
    # Phase 3: Crawl
    print("[Phase 3] Running Katana crawler...")
    run_katana(target_url, session_data)
    
    # Phase 4: Clean data
    print("[Phase 4] Processing crawler data...")
    subprocess.run(["python", "scripts/clean_crawl_data.py",
                   "--input", f"{workspace}/crawled_anonymous.jsonl",
                   "--output", f"{workspace}/targets.txt"])
    
    # Phase 4.5: Map attack surface
    print("[Phase 4.5] Mapping attack surface...")
    subprocess.run(["python", "scripts/map_attack_surface.py",
                   "--targets", f"{workspace}/targets.txt",
                   "--fingerprint", f"{workspace}/fingerprint.json",
                   "--output", f"{workspace}/attack_surface.json"])
    
    # Phase 5: Multi-agent testing
    print("[Phase 5] Launching 6 parallel agents...")
    subprocess.run(["python", "scripts/prepare_agent_findings.py",
                   "--targets", f"{workspace}/targets.txt",
                   "--fingerprint", f"{workspace}/fingerprint.json",
                   "--output", f"{workspace}/findings"])
    
    # Phase 5.5: Attack chain analysis
    print("[Phase 5.5] Analyzing attack chains...")
    subprocess.run(["python", "scripts/analyze_chains.py",
                   "--findings", f"{workspace}/findings",
                   "--output", f"{workspace}/attack_chains.json"])
    
    # Phase 5.6: Verify evidence
    print("[Phase 5.6] Verifying vulnerability evidence...")
    subprocess.run(["python", "scripts/verify_findings.py",
                   "--findings", f"{workspace}/findings"])
    
    # Phase 6: Generate reports
    print("[Phase 6] Generating final reports...")
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    subprocess.run(["python", "scripts/generate_report.py",
                   "--findings-dir", f"{workspace}/findings",
                   "--output-json", f"{workspace}/report_{timestamp}.json",
                   "--output-html", f"{workspace}/report_{timestamp}.html",
                   "--output-docx", f"{workspace}/report_{timestamp}.docx"])
    
    print(f"✓ Reports generated in {workspace}/report_result/")

# Usage
run_standard_pentest(
    target_url="https://example.com",
    auth_statement="Written authorization obtained for full-scope testing",
    account_info={"username": "testuser", "password": "from_env"}
)

Multi-Account Privilege Escalation Testing

"""
Test for privilege escalation and horizontal authorization bypass
using multiple accounts with different permission levels
"""
def run_multiuser_pentest(target_url, accounts):
    workspace = "workspace"
    
    # Extract sessions for all accounts
    sessions = {}
    for role, account in accounts.items():
        print(f"[Phase 2.{role}] Login as {role}...")
        sessions[role] = extract_session(target_url)
        with open(f"{workspace}/sessions/{role}_session.json", "w") as f:
            json.dump(sessions[role], f)
    
    # Crawl with each role
    for role, session in sessions.items():
        print(f"[Phase 3.{role}] Crawling as {role}...")
        run_katana(target_url, session, output_prefix=role)
    
    # Merge crawl results
    subprocess.run(["python", "scripts/merge_crawl_results.py",
                   "--inputs", f"{workspace}/*_crawled.jsonl",
                   "--output", f"{workspace}/targets.txt"])
    
    # Continue with standard workflow...
    # Phase 5 agents will automatically test for IDOR/privilege escalation
    # using the multiple session data

# Usage
run_multiuser_pentest(
    target_url="https://example.com",
    accounts={
        "admin": {"username": "admin", "password": "from_env"},
        "user": {"username": "normaluser", "password": "from_env"}
    }
)

Report-Only Generation

"""
Generate reports from existing findings without re-scanning
Useful when you need to regenerate reports after manual review
"""
def generate_reports_only(workspace="workspace"):
    from datetime import datetime
    
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    
    # Phase 5.5: Re-analyze attack chains
    subprocess.run(["python", "scripts/analyze_chains.py",
                   "--findings", f"{workspace}/findings",
                   "--output", f"{workspace}/attack_chains.json"])
    
    # Phase 5.6: Re-verify findings
    subprocess.run(["python", "scripts/verify_findings.py",
                   "--findings", f"{workspace}/findings"])
    
    # Phase 6: Generate reports
    subprocess.run(["python", "scripts/generate_report.py",
                   "--findings-dir", f"{workspace}/findings",
                   "--fingerprint", f"{workspace}/fingerprint.json",
                   "--output-json", f"{workspace}/report_{timestamp}.json",
                   "--output-html", f"{workspace}/report_{timestamp}.html",
                   "--output-docx", f"{workspace}/report_{timestamp}.docx"])
    
    # Validate report formats
    validate_reports(workspace, timestamp)

def validate_reports(workspace, timestamp):
    import os
    
    json_path = f"{workspace}/report_{timestamp}.json"
    html_path = f"{workspace}/report_{timestamp}.html"
    docx_path = f"{workspace}/report_{timestamp}.docx"
    
    assert os.path.exists(json_path), "JSON report missing"
    assert os.path.exists(html_path), "HTML report missing"
    assert os.path.exists(docx_path), "DOCX report missing"
    
    with open(json_path) as f:
        report_data = json.load(f)
        assert "findings" in report_data
        assert "summary" in report_data
    
    print("✓ All report formats validated")

Troubleshooting

Katana Crawler Issues

Problem: Crawler returns empty results or finishes in <20 seconds

# Solution: Verify session cookies are valid
import json

with open("workspace/sessions/session.json") as f:
    session = json.load(f)
    
# Check cookie expiration
for cookie in session["cookies"]:
    if "expires" in cookie:
        print(f"{cookie['name']}: expires {cookie['expires']}")

# Re-extract session if cookies expired
session = extract_session(target_url)

Problem: Crawler times out or hangs

# Solution: Reduce crawl depth and concurrency
python scripts/run_katana.py \
    --url https://example.com \
    --depth 3 \
    --concurrency 5 \
    --timeout 600

Multi-Agent Coordination

Problem: Agents not finding vulnerabilities

# Solution: Check that skeleton findings were properly generated
import os

findings_dir = "workspace/findings"
agents = ["auth", "injection", "logic", "upload", "api", "info"]

for agent in agents:
    skeleton_path = f"{findings_dir}/{agent}_findings.json"
    if not os.path.exists(skeleton_path):
        print(f"Missing skeleton for {agent} agent")
        # Regenerate skeletons
        subprocess.run(["python", "scripts/prepare_agent_findings.py",
                       "--targets", "workspace/targets.txt",
                       "--fingerprint", "workspace/fingerprint.json",
                       "--output", findings_dir])
        break

Problem: Agents marking everything as "Potential" without confirmation

Reminder for AI agents:
- Must attempt actual exploitation, not just theory
- Require HTTP request/response evidence for "Confirmed" status
- Try 2-3 bypass techniques on WAF/validation failures
- Mark as "Potential" only if technical constraints prevent confirmation

Report Generation Failures

Problem: Report missing sections or malformed

# Solution: Validate findings structure before report generation
def validate_findings_structure(findings_dir):
    import glob
    
    for finding_file in glob.glob(f"{findings_dir}/*_findings.json"):
        with open(finding_file) as f:
            data = json.load(f)
            
        required_fields = ["agent_name", "findings", "summary"]
        for field in required_fields:
            assert field in data, f"Missing {field} in {finding_file}"
            
        for finding in data["findings"]:
            assert "title" in finding
            assert "severity" in finding
            assert "status" in finding  # Confirmed, Potential, or False Positive
            assert "evidence" in finding
            
    print("✓ All findings files valid")

validate_findings_structure("workspace/findings")

Session Extraction Issues

Problem: Browser doesn't launch or session not captured

# Solution: Use explicit browser path and user data directory
from playwright.sync_api import sync_playwright

def extract_session_robust(target_url):
    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=False,
            executable_path="/usr/bin/chromium",  # Adjust for your system
            args=["--disable-blink-features=AutomationControlled"]
        )
        
        context = browser.new_context(
            viewport={"width": 1920, "height": 1080},
            user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)..."
        )
        
        page = context.new_page()
        page.goto(target_url)
        
        print("Please log in manually. Press Enter when done...")
        input()
        
        # Comprehensive credential extraction
        cookies = context.cookies()
        storage = page.evaluate("() => Object.assign({}, localStorage)")
        session_storage = page.evaluate("() => Object.assign({}, sessionStorage)")
        
        # Capture auth headers from network traffic
        auth_headers = {}
        def handle_response(response):
            if "authorization" in response.request.headers:
                auth_headers["Authorization"] = response.request.headers["authorization"]
        
        page.on("response", handle_response)
        page.reload()
        page.wait_for_load_state("networkidle")
        
        browser.close()
        
        return {
            "cookies": cookies,
            "localStorage": storage,
            "sessionStorage": session_storage,
            "headers": auth_headers
        }

Best Practices

1. Always use separate workspaces for different targets to avoid cross-contamination
2. Verify authorization documentation before starting any test
3. Test on staging/UAT environments when possible, not production
4. Review findings manually before delivering reports to clients
5. Keep vibe-pentest updated using git pull to get latest detection techniques
6. Use multiple accounts to thoroughly test authorization controls
7. Document custom test data created during testing for cleanup

Additional Resources

• 3-minute tutorial video: bilibili.com/video/BV1RiGX6rESQ/
• Sub-agent setup guide: See sub_agent.md in repository
• Katana documentation: github.com/projectdiscovery/katana