# Challenge Pipeline Audit

> Challenge Pipeline Audit — Sentinel Standard

- Skill: `nickgallick/challenge-pipeline-audit` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nickgallick/challenge-pipeline-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nickgallick/challenge-pipeline-audit/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: nickgallick (https://skillmd.com/u/nickgallick)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/nickgallick/challenge-pipeline-audit

---

# Challenge Pipeline Audit — Sentinel Standard

## What the Pipeline Is
Challenges go through a 14-state lifecycle from Gauntlet creation to live competition.

**States**: draft → draft_failed_validation | draft_review → needs_revision | approved_for_calibration → calibrating → passed | flagged → passed_reserve | queued | active → quarantined | retired | archived

## Pipeline APIs to Test

### Intake (Gauntlet → Platform)
- `POST /api/challenges/intake`
- Auth: `Bearer a86c6d887c15c5bf259d2f9bcfadddf9`
- Required fields: family, weight_class, format, name, description, prompt, starter_code, hidden_tests, difficulty_profile, scoring_rubric, calibration_expectations, evidence_map

### Forge Review (Admin)
- `GET /api/admin/forge-review` — list pending review queue
- `POST /api/admin/forge-review` — submit verdict (approve/reject/needs_revision)

### Inventory (Operator Decision)
- `GET /api/admin/inventory` — list with advisories
- `POST /api/admin/inventory` — submit decision (publish_now/hold_reserve/queue_for_later/mutate_before_release/quarantine/reject)

### Quality Enforcement
- `GET /api/cron/challenge-quality` — runs CDI check
- `GET /api/admin/challenge-quality` — quality status

### Calibration
- `POST /api/admin/calibration` — run_synthetic / run_full / run_forced_real / mutate

## What to Audit

### Availability Check
- Do all pipeline API routes return responses (not 500)?
- Is challenge_bundles table accessible? (⚠️ Migration 00024 may be partially applied)
- Are forge-review and inventory routes protected (return 401 unauthed)?

### Data Integrity
- Do challenges in the DB have valid pipeline_status values?
- Are activation snapshots created when challenges go active?
- Is the CDI score present on calibrated challenges?

### UI Access
- Can an admin operator access the pipeline management UI?
- Are challenge states visible in /admin/challenges?
- Is there a way to manually trigger pipeline transitions from the UI?

### Error Handling
- What happens when you POST to /api/challenges/intake with invalid data?
- What happens when migration 00024 is not applied (challenge_bundles missing)?
- What does the error look like to an operator?

## Known Issues (as of 2026-03-29)
- ⚠️ Migration 00024 partially applied — challenge_bundles table may not exist
  - All intake/forge-review/inventory routes will 500 until fixed
  - Forge needs to re-call migration runner with correct Bearer header
  - This is a P0 for the pipeline — nothing can flow through until resolved

## Calibration System
- Synthetic runner: fast automated calibration
- Real LLM runner: actual model evaluation (more expensive)
- Policy: daily=synthetic only, standard=synthetic+optional real, featured/prize=both required
- Calibration results stored in challenge_calibration_results table
- CDI (Challenge Difficulty Index) computed from calibration runs

## Quality Enforcement
- Runs every 15 minutes via cron
- Checks CDI, judge divergence, score spread
- Auto-flags challenges below quality threshold
- Auto-quarantines on critical failures
- Results via /api/admin/challenge-quality

## Activation Gate Requirements
A challenge can only go active if:
- has_objective_tests = true
- prompt present
- description present  
- format present
- time_limit present
- calibration_status = passed

## Test Checklist
- [ ] POST /api/challenges/intake with valid bundle → expect 200 or appropriate error
- [ ] GET /api/admin/forge-review → expect queue (may be empty)
- [ ] GET /api/admin/inventory → expect list with advisories
- [ ] GET /api/cron/challenge-quality → expect JSON result
- [ ] Verify challenge_bundles table exists (migration 00024)
- [ ] Check active challenges have activation_snapshot
- [ ] Verify /api/challenges returns real challenges with correct fields