# Trailblaze Validate Oob

> Use when validating or evaluating Trailblaze's out-of-box (OOB) user experience — does what the `trailblaze` skill claims actually match the real installed CLI? Triggers on requests to "validate the Trailblaze skill", "test the Trailblaze OOB experience", "evaluate Trailblaze UX", "check if the skill matches the CLI", "audit the Trailblaze CLI for new-user friction", or running an OOB regression check after the framework changes.

- Skill: `block/trailblaze-validate-oob` (Agent Skill)
- Install (CLI): `npx skillmds@latest add block/trailblaze-validate-oob`
- Raw SKILL.md: https://api.skillmd.com/api/skills/block/trailblaze-validate-oob/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: block (https://skillmd.com/u/block)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/block/trailblaze-validate-oob

---


# Validating Trailblaze's out-of-box experience

This skill encodes the methodology for honestly evaluating whether
Trailblaze's user-facing CLI matches what a fresh agent / new user
would intuitively expect. It's the framework's own UX regression
test, run by an agent simulating cold-start use.

The goal is to catch counter-intuitive defaults, broken claims in the
`trailblaze` skill, and friction that the team has gone CLI-blind on.

## When to run a validation pass

- After any change to the `trailblaze` CLI surface (commands, flags,
  default behavior)
- After any change to the `trailblaze` Claude skill
  (the sibling `trailblaze/SKILL.md`) or its references
- Periodically (monthly?) even with no changes, to catch drift
- When a new rung of the adoption ladder gets added to the skill

## What "good" looks like

After a validation pass, **a fresh agent following only the
`trailblaze` skill should be able to complete the scenario with zero
surprises**. Specifically:

- Every command the skill says to run actually exists with the
  documented flags and argument shape
- Every output the skill describes matches what the CLI actually
  prints
- The basic loop (drive → save → replay → inspect) works end-to-end
  with no undocumented gotchas
- Failure modes the skill mentions are real failure modes; the
  recovery steps actually recover
- Errors the user might hit have clear `hint:` lines pointing at the
  fix
- No "workaround" paragraphs in the skill that exist to paper over
  CLI behavior the user wouldn't naturally guess

If the validation surfaces something the agent couldn't have guessed
from the skill alone, that's either a skill update OR a CLI fix —
and the latter is usually preferable.

## Methodology

### Step 1: Install the binary fresh

Use the actual end-user install path, NOT `./trailblaze` from the
repo (which goes through Gradle on every call and isn't what users
hit):

```bash
./scripts/install-trailblaze-source.sh
```

The script builds the uber JAR and installs it under
`~/.trailblaze/install/` with a symlink onto the system PATH. The
specific symlink location depends on your platform (homebrew on
macOS, `/usr/local/bin/` on Intel macs / Linux, …) — defer to the
script's stdout for the exact path rather than memorizing one. Then
verify:

```bash
which trailblaze         # confirms which binary the shell will pick up
trailblaze --version     # records the version under test (capture this in the report)
```

If the script fails (missing JDK, Gradle error, symlink-permission
failure), **that's itself a finding**: file it as an "OOB blocker"
and spawn a CLI fix chip *before* trying to run the validation —
don't shim around it from inside the skill.

### Step 2: Spawn a fresh-context subagent

The subagent must have **no prior Trailblaze knowledge**. Their only
source of truth is the `trailblaze` skill file. They must not infer
commands from the repo's source code or from any other docs.

The right tool for this is a general-purpose agent (read-only is
sufficient since the validation only runs `--help` and discovery
commands, never destructive actions).

If the subagent can't proceed at this step — no device connected,
target app not installed, network or auth required, etc. — that's a
gap too: an OOB experience that requires undocumented setup is itself
a finding. Record what setup was needed and what error the subagent
hit, file it the same way as an OOB blocker, and resume the
validation once the setup gap is closed.

### Step 3: Give the subagent a concrete scenario

A scenario must:

- Be a realistic user task ("drive an Android emulator", "save a
  flow as a trail", "replay a trail and inspect what happened")
- Be doable using ONLY commands the skill describes
- Have a clear "I would now act on a device" stopping point — the
  validation is **read-only**; don't actually drive devices or
  modify state

Scenarios that have worked well:

- **Rung 1**: "Drive a device. List devices, snapshot one, look at
  the toolbox to find available actions, identify how you'd act on
  an element. Stop before actually executing the action."
- **Rung 2**: "Save the current session as a `.trail.yaml`. Then
  replay one. Then generate a report. Then look up past results."
- **Rung 3**: (when written) "Compose your own agent surface — write
  a custom typed tool, install a trailmap, curate what tools the
  agent sees."

### Step 4: Subagent prompt template

The validation prompt should always:

1. Tell the subagent it's a Claude Code session that just loaded the
   skill — no prior Trailblaze knowledge
2. Specify the **installed `trailblaze` binary on PATH** as the test
   surface (NOT `./trailblaze`)
3. Give the concrete scenario
4. Instruct them to follow ONLY what the skill says; gaps are gaps
5. Require structured output: **Inaccuracies**, **Gaps**,
   **Ambiguities**, **What works**, **Suggestions**
6. Cite quoted skill text and actual CLI output side-by-side for
   each finding

### Step 5: Triage findings

For each finding the subagent reports:

| Finding type | Right action |
|---|---|
| Skill claims X; CLI does Y | Update the skill OR spawn a CLI fix chip. Lean toward CLI fix if X is the more intuitive design. |
| Skill is silent; agent had to guess | Add to the skill. Note explicitly if it's a workaround vs intentional behavior. |
| Skill is ambiguous; agent picked one interpretation | Tighten the skill text. |
| Skill claim worked exactly as described | Keep — note what worked so it doesn't accidentally regress. |

**Bias toward fixing the CLI**, not the skill. If a fresh agent
guessed differently than the current CLI, the agent is usually
modeling the intuitive design — and the CLI is the thing diverging
from intuition.

### Step 6: Run a second pass after fixes

After applying fixes (either in the skill or via spawned CLI chips),
**run the validation again** with a different fresh-context
subagent. Two passes catch issues the first pass's fixes
introduced and confirm the fixes hold against another cold reader.

Don't reuse the same subagent — context contamination defeats the
"fresh new user" model.

## What to NOT do

- **Don't run validation against `./trailblaze`** — that's the
  Gradle-wrapped dev launcher, not the end-user binary. Findings
  there ("first call triggers a Gradle build", "needs
  `local.properties`") are dev-environment concerns, not OOB UX
  problems.
- **Don't pre-bias the subagent** with answers. Don't say "verify
  that `tap` is the tool name" — let them discover it from the skill
  and the CLI.
- **Don't have the subagent execute destructive actions.** No
  driving devices, no modifying state, no committing files. Read-only
  validation only — `--help`, `device list`, `snapshot`, etc.
- **Don't ignore a subagent finding because "the current CLI design
  is what it is".** That's how counter-intuitive designs calcify.

## What a healthy validation report looks like

A useful report has all of:

- **Inaccuracies** with quoted skill text + actual CLI output for
  each
- **Verified claims** — explicitly list what held up, so the team
  knows what's safe to keep
- **Gaps** the subagent had to guess past
- **Suggestions** — concrete edits to skill OR concrete CLI fixes
- Cap around 500 words; brevity forces precision

If the report is just "looks fine to me," the methodology wasn't
adversarial enough — re-prompt with a more pointed scenario.

## After validation: distribution

Findings flow three ways — the first one is non-optional, the other
two depend on what surfaced:

1. **Durable record of the pass itself.** Before spawning any chips
   or editing any skill, post a short summary to a persistent surface
   — a GitHub issue on the project's issue tracker (preferred), or
   the relevant team chat thread, or a tracked markdown file. Include:
   date, the rung exercised, the `trailblaze --version` you captured
   in Step 1, a one-line outcome ("clean", "N findings, M chipped"),
   and links to any chips or follow-up PRs. The chat transcript of
   the validation is *not* a durable artifact — without this, "did
   we last validate after the X CLI change?" is unanswerable.
2. **Skill updates** — edit the sibling `trailblaze/SKILL.md`
   directly, run the sensitive-terms scanner, commit, PR.
3. **CLI fix chips** — spawn a focused chip for the framework
   behavior change, with the validation finding as motivation in
   the chip's prompt.

Skill updates and CLI chips should reference the validation pass
that surfaced the finding (the issue/Slack URL from #1), so future
contributors can trace why a change was made.