# Scaffold Trial

> Orchestrate scaffold smoke testing across presets. Dispatches parallel tester agents, collects reports, synthesizes findings, and files Linear issues. Use when scaffold trial, smoke test presets, or /scaffold-trial are mentioned.

- Skill: `outfitter-dev/scaffold-trial` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add outfitter-dev/scaffold-trial`
- Raw SKILL.md: https://api.skillmd.com/api/skills/outfitter-dev/scaffold-trial/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: outfitter-dev (https://skillmd.com/u/outfitter-dev)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/outfitter-dev/scaffold-trial

---


# Scaffold Trial

Orchestrate agent-driven scaffold smoke testing across all presets.

## Workflow

### Step 1: Parse Arguments

If `$ARGUMENTS` is provided:

- `all` → test all 7 presets: cli, library, full-stack, minimal, basic, mcp, daemon
- Comma-separated list → test only those presets (validate each is a valid preset name)
- Empty → offer smart selection (see below)

**Smart selection** (when no arguments):

1. Run `git log --oneline -20` to see recent changes
2. Map changed packages to affected presets:
   - `@outfitter/cli` changes → cli, full-stack
   - `@outfitter/mcp` changes → mcp, full-stack
   - `@outfitter/daemon` changes → daemon
   - `packages/presets` changes → all presets
   - `apps/outfitter` (init command) changes → all presets
3. Present options to user via AskUserQuestion:
   - Suggested subset based on changes
   - "all" option always available
   - Individual preset selection

### Step 2: Prune Old Runs

Delete `.test/scaffolds/` entries older than 7 days:

```bash
find .test/scaffolds/ -maxdepth 1 -type d -mtime +7 -exec rm -rf {} +
```

Create `.test/scaffolds/` if it doesn't exist.

### Step 3: Create Run Directory

Generate run ID: `YYYYMMDDTHHmmss-trial-<uuid-v7>`

Create directory structure:

```
.test/scaffolds/<run-id>/
  <preset>/     # One subdir per selected preset
```

### Step 3b: Write Run Directory Marker

Write the run directory absolute path to `.test/scaffolds/.current-run-dir` so PostToolUse hooks in subagent processes can discover it (env vars cannot be injected into hook subprocesses):

```bash
echo "<absolute-path-to-run-dir>" > .test/scaffolds/.current-run-dir
```

This file is the fallback for `OUTFITTER_SCAFFOLD_TRIAL_RUN_DIR`. The `log-telemetry.ts` hook checks the env var first, then falls back to reading this file.

### Step 4: Initialize Manifest

Write `manifest.json` to the run directory:

```json
{
  "runId": "<run-id>",
  "startedAt": "<ISO 8601>",
  "presets": {
    "<preset>": {
      "status": "pending",
      "reportPath": ".test/scaffolds/<run-id>/<preset>/report.json"
    }
  }
}
```

### Step 5: Dispatch Tester Agents

For each preset, launch a `scaffold-tester` agent:

```
Agent tool call:
  subagent_type: "scaffold-tester"
  run_in_background: true
  prompt: |
    Test the "<preset>" scaffold preset.

    Run directory: <absolute-path-to-run-dir>/<preset>/
    Preset: <preset>
    Run ID: <run-id>
    Report path: <absolute-path-to-run-dir>/<preset>/report.json

    Set OUTFITTER_SCAFFOLD_TRIAL_RUN_DIR=<absolute-path-to-run-dir> for telemetry.

    Scaffold command (run from repo root <repo-root>):
    bun ./apps/outfitter/src/cli.ts init <absolute-path-to-run-dir>/<preset>/project --name scaffold-e2e-<preset> --preset <preset> --yes --skip-git --skip-commit

    Follow the scaffold-testing skill workflow completely. Write report.json when done.
```

Launch ALL preset agents in a single message (parallel dispatch). Record each agent ID in the manifest immediately.

Update manifest after each dispatch:

```json
{ "status": "running", "agentId": "<agent-id>" }
```

### Step 6: Track Completion

As agents complete (you'll be notified automatically — do NOT poll):

- Update manifest status to `completed` or `errored`
- If an agent errors, record the error message in the manifest
- Continue waiting for remaining agents

Timeout: If an agent hasn't completed after 10 minutes, note it as timed out in the manifest.

### Step 7: Handle Failures

For any agent that crashes or times out:

- Mark as `errored` in manifest with error details
- Continue with available reports — don't abort the whole run

### Step 8: Synthesize

Once all agents have completed (or timed out):

1. Load the `scaffold-reporting` skill
2. Follow its workflow to generate `summary.json` and `summary.md`
3. File Linear issues for blocking/degraded findings

### Step 9: Present Results

Show the user a formatted results table:

```
Scaffold Trial: <run-id>
═══════════════════════════════════════

Preset       Status    Overall  Agent ID
───────────────────────────────────────
cli          PASS      8/10     abc123
library      PASS      7/10     def456
full-stack   FAIL      5/10     ghi789
minimal      PASS      6/10     jkl012
mcp          ERROR     --       mno345
daemon       PASS      7/10     pqr678

Score Summary (mean / min / max):
  Agent Readiness:    7.2 / 5 / 9
  Setup Friction:     7.8 / 6 / 9
  Doc Completeness:   7.0 / 4 / 9
  Error Clarity:      6.5 / 3 / 8
  Type Correctness:   8.0 / 6 / 10

Linear Issues Filed: 2
  OS-xxx: Missing verify:ci in minimal/daemon
  OS-yyy: Build failure in full-stack preset

Run directory: .test/scaffolds/<run-id>/
```

Include agent IDs so the user can resume individual agents for follow-up.

## References

- `references/manifest-schema.md` — Manifest JSON structure