Reality Check Sparta

name: reality-check-sparta description: Adversarial data quality assessment for the SPARTA QRA pipeline with self-correction loop. Actively hunts for flaws using multiple fresh verification techniques. Models The Aerospace Corporation's SPARTA Framework. triggers: reality check sparta, sparta check, sparta quality, adversarial check, sparta fidelity, brandon review

provides: - reality-check-sparta composes: [, task-monitor]

reality-check-sparta

ADVERSARIAL data quality assessment for the SPARTA QRA pipeline with self-correction loop. Actively hunts for flaws using multiple fresh verification techniques.

Client Knowledge

This skill models The Aerospace Corporation's SPARTA Framework:

SPARTA Structure

• 216 Techniques: Categorized by tactic (REC=Reconnaissance, EX=Execution, IA=Initial Access, etc.)
• 91 Countermeasures: Security controls mapped to techniques
• Cross-references: MITRE ATT&CK, NIST 800-53, D3FEND, CWE, ESA SPACE-SHIELD

Technique Categories

Philosophy

• Any mismatch is a flaw to investigate, not "compensate for"
• Trust nothing, verify everything - high scores might be gamed
• Use fresh techniques each iteration - not just re-running same checks
• Model the client - verify against original SPARTA data source
• A PASS is hard to earn - multiple warnings = FAIL

Self-Correction Loop

The skill implements an iterative self-correction workflow:

┌─────────────────────────────────────────────────────────────────┐
│                    SELF-CORRECTION LOOP                         │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  1. CHECK (adversarial)                                        │
│     ├── Database sampling + file inspection                    │
│     ├── SPARTA source fidelity (verify against Excel)          │
│     ├── URL/file alignment detection                           │
│     ├── Fresh URL fetch via httpx                              │
│     └── Verbatim grounding verification                        │
│                         │                                       │
│                         ▼                                       │
│  2. ANALYZE (identify issues)                                  │
│     ├── Categorize by severity (CRITICAL > HIGH > MEDIUM)      │
│     ├── Suggest root causes                                    │
│     └── Map to pipeline owners                                 │
│                         │                                       │
│                         ▼                                       │
│  3. SUGGEST FIXES                                              │
│     ├── Specific actionable fixes per issue category           │
│     └── Track convergence over time                            │
│                         │                                       │
│                         ▼                                       │
│  4. APPLY FIXES (human/pipeline)                               │
│                         │                                       │
│                         ▼                                       │
│  5. RE-CHECK (fresh techniques)                                │
│     ├── Use DIFFERENT verification methods                     │
│     ├── Cross-reference against SPARTA website                 │
│     └── Fresh fetch URLs to compare                            │
│                         │                                       │
│                         ▼                                       │
│  6. CONVERGENCE ANALYSIS                                       │
│     ├── Track issue counts over iterations                     │
│     ├── IMPROVING / STABLE / REGRESSING status                 │
│     └── Store learnings in /memory                             │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Quick Start

cd /home/graham/workspace/experiments/pi-mono/.pi/skills/reality-check-sparta

# Run adversarial check with fix suggestions
./run.sh check --run-id run-recovery-verify --samples 20

# Self-correction loop (check + convergence)
./run.sh iterate --run-id run-recovery-verify

# Check convergence trend
./run.sh convergence

# Full deep check
./run.sh check --run-id run-recovery-verify --full --store

Fresh Verification Techniques

Each iteration uses different methods to avoid blind spots:

What It Checks (Adversarially)

1. SPARTA Source Fidelity (NEW)

• Verifies DB accurately represents original SPARTA Excel
• Checks technique/countermeasure counts (216/91)
• Validates ID format matches SPARTA convention
• Verifies cross-reference column coverage

2. URL/File Integrity

• Detects mismatched downloads (file contains wrong technique)
• >5% mismatch = FAIL
• Fresh-fetches URLs to compare against cached content

3. Verbatim Grounding Verification

• Actually checks if QRA answers match source text
• 20-char phrase matching
• Detects hallucination patterns

4. QRA Structure Integrity

• Empty/null answers
• Orphan QRAs (no relationship)
• Duplicate questions

5. Coverage Gap Analysis

• Weak frameworks (avg grounding < 0.85)
• Low relationship coverage

Fix Suggestions

The skill suggests specific fixes for each failing check:

🔴 [CRITICAL] url_file_alignment: FAIL
   Description: Files downloaded for MITRE ATT&CK URLs contain wrong technique content
   Root Cause: Likely redirect handling, hash collision, or race condition
   Owner: fetch/download logic in SPARTA pipeline
   Suggested Fixes:
      1. Check download function for proper redirect following
      2. Add URL->content validation in download pipeline
      3. Re-download mismatched URLs individually
      4. Implement checksumming for downloaded files

Convergence Analysis

Track issue resolution over time:

./run.sh convergence

# Output:
CONVERGENCE ANALYSIS
Status: IMPROVING
Message: Issues decreased from 25 to 18

Recent History:
  2026-02-06T08:42:53: 25 issues
  2026-02-06T09:15:00: 22 issues
  2026-02-06T10:30:00: 18 issues

Commands

QRA Convergence Model

QRA quality improvement follows the same dynamics as model training convergence:

Convergence Rules:

1. Issue count MUST decrease cycle over cycle (like loss decreasing)
2. 3 consecutive regressions = stalled → human intervention needed
3. Plateau = prompt ceiling → redesign with /prompt-lab
4. NEVER lower thresholds to game the curve
5. Track metrics via convergence_state.json

Use sparta-review converge for the full autonomous loop.

Dynamic Thresholds (Annealing Schedule)

Brandon decides thresholds based on corpus size - like annealing in model training:

• Early (small corpus): Be lenient, allow learning
• Middle (growing): Gradually tighten standards
• Late (near target): Apply strictest standards

The assessment now includes a continue_decision field:

• CONTINUE - Quality acceptable for current phase, keep generating
• STOP_AND_FIX - Quality below phase threshold, run auto-fix

Static Thresholds (Non-Annealed)

Memory Integration

Findings stored with adversarial framing:

# After check with --store:
# Stores: "SPARTA reality check run-X: 25 ISSUES FOUND"
# With: "ACTION REQUIRED: Investigate root causes"

# Query past findings:
./run.sh history

Brandon Bailey Persona

The skill uses the Brandon Bailey persona for domain expert quality review. Brandon Bailey is the creator of SPARTA at The Aerospace Corporation.

Persona Integration

# Run Brandon Bailey review (checks space terminology)
./run.sh check --run-id run-recovery-verify --brandon-review

# Full review with prompt optimization recommendations
./run.sh check --run-id run-recovery-verify --brandon-review --full

Grading Scale

Required Space Terminology

Brandon requires every QRA answer to include space-specific terms:

• Segment Context: ground segment, link segment, space segment
• Assets: satellite, spacecraft, payload, bus, ground station
• Communications: RF, SATCOM, uplink, downlink, telemetry, TT&C
• Threats: jamming, spoofing, signal interference, ASAT
• Standards: CCSDS, SpaceWire, MIL-STD

Persona Files

• BRANDON_BAILEY_PERSONA.md - Full persona definition
• brandon_bailey_persona.yaml - Create-persona manifest
• LESSONS_LEARNED.md - Session learnings (F → A+)

Related Skills