0x-Shashi Audit Scoring
Purpose
This directory provides the quantitative scoring framework for measuring audit quality. Use these metrics to objectively evaluate performance, track improvement over time, and identify areas needing attention.
Available Files
| File | Description |
|---|---|
| AUDIT_SCORING.md | Complete scoring system — detection, precision, severity accuracy, coverage, efficiency metrics, composite score formula, reward schema, tracking templates, and industry benchmarks |
Core Metrics at a Glance
| Metric | Weight | What It Measures |
|---|---|---|
| Detection Score | 35% | Vulnerabilities correctly identified vs. total real vulnerabilities |
| Precision Score | 25% | Valid findings vs. total findings submitted (false positive rate) |
| Severity Accuracy | 15% | Correct severity classification vs. total findings |
| Coverage Score | 15% | Functions/entry points audited vs. total codebase |
| Efficiency Score | 10% | Weighted findings produced per hour spent |
Composite Score =
(0.35 × Detection) + (0.25 × Precision) + (0.15 × Severity) + (0.15 × Coverage) + (0.10 × Efficiency)
Severity Weights for Efficiency Scoring
These weights connect the scoring system to the severity classification:
| Severity | Points | Reference |
|---|---|---|
| Critical | 10 | Escalation required (not in standard severity files) |
| High | 5 | high-severity.md |
| Medium | 2 | medium-severity.md |
| Low | 1 | low-severity.md |
| Informational | 0.5 | Best-practice suggestions |
| Gas | 0 | gas-optimizations.md |
How to Use
- After an audit → Fill out the Score Card Template in AUDIT_SCORING.md
- Classify findings → Use severity/ files + severity-scoring decision tree
- Track monthly → Use the Monthly Score Tracking template
- Identify gaps → Category-specific scores highlight weak areas
- Improve → Low category scores → update checklists and patterns
Related Skills
- Severity Classification — HIGH / MEDIUM / LOW / GAS finding databases
- Severity Scoring Decision Tree — How to assign severity levels
- Feedback Loop — Scores feed back into skill improvement
- Audit Report Templates — Report structure with severity sections
- Prompt Evolution — Higher-scoring prompts get promoted
Prerequisites
Scoring requires completed audit findings with severity classifications. The Severity Classification skill MUST be applied before scoring.
Validation
To verify scoring accuracy, compare computed composite scores against known benchmarks:
# Example composite score calculation
detection = 0.85 # 85% of real vulns found
precision = 0.80 # 80% valid findings
severity_acc = 0.90 # 90% correct severity
coverage = 0.75 # 75% codebase covered
efficiency = 0.70 # Weighted findings per hour
composite = (0.35 * detection + 0.25 * precision + 0.15 * severity_acc + 0.15 * coverage + 0.10 * efficiency)
print(f"Composite Score: {composite:.2f}") # Expected: 0.81
# Score thresholds for audit quality tiers
tiers:
elite: 0.90+ # Top-tier competitive auditor
proficient: 0.75+ # Solid professional auditor
developing: 0.60+ # Learning auditor
needs_work: <0.60 # Consider additional training
# Validate scoring data integrity
python scripts/quality-check.py skills/scoring/SKILL.md
Behavior Guidelines
- Detection and Precision scores are required for every engagement
- Coverage tracking is optional for quick scans but MUST be included in full audits
- Efficiency scoring should be used for self-improvement, never to rush audits
References
- Scoring References - Industry benchmarks and calibration data