Trial Run

/trial-run, trial planner and logger

Two modes. Ask which applies if it isn't obvious from the conversation.

A trial run is the only thing that produces a real score. Impressions of how a run looked are not data, and the report's results section needs the numbers. See the test rungs in .claude/rules/workflow.md.

The discipline this skill exists to enforce

One run proves almost nothing. The marker digits, the colour-to-row mapping, the book's horizontal position and the robot's starting orientation all re-randomise on every world load. A single successful run may only mean the seed was favourable.

So every plan states a reload count, and every log states which seed conditions applied.

Mode 1: plan, before the run

1. Ask what's being tested and what would count as passing. A test without a pre-declared numeric criterion is a demo, not a test.

2. Write the procedure:

   • Which subsystem or which full-run stage
   • How many world reloads. 20 for anything perception-related, 10 for grasp attempts. Not 3
   • Which requested column and colour on each run, varied across the set
   • What gets measured and how it's recorded
   • What gets changed between runs, one variable at a time

3. State the pass criterion numerically. Starting points, from .claude/agents/qa.md:

   • Column detection: correct digit on 20 of 20 world reloads
   • Book detection: correct colour and row on 20 of 20 reloads, all four colours and all four rows represented
   • Annotated images: present, timestamped, correctly boxed, in 20 of 20 runs
   • Navigation: reaches the standoff pose 20 of 20, zero collisions
   • Grasp: book lifted clear in 8 of 10 attempts, broken out by row
   • Gentle place: released under control in 8 of 10
   • Recovery: each path deliberately triggered and observed at least once

4. Pre-run checklist, mandatory, from .claude/rules/workflow.md:

   • colcon build clean, workspace sourced
   • simulation.launch.py running first, world fully loaded
   • Launch with the exact organiser command, both arguments passed
   • erc_images/ writable and cleared of stale images
   • results/ writable, trial_logger running
   • Both scored topics confirmed live with ros2 topic echo
   • World reloaded since the last run, so this is a new seed
   • Requested column and colour recorded before the run

5. Note the two traps:

   • Stale images. Images produced outside the trial run are explicitly not accepted, and a leftover file from yesterday is indistinguishable from one. Clear the folder.
   • Repeating a seed. Running twice without reloading is one test, not two.

Mode 2: log, after the run

1. Ask for the real point breakdown, line by line against the official criteria, not an impression. "13 of 19: both IDs scored, grasp on the second attempt, dropped rather than gently placed, one collision on retreat" is data. "The run went well" is not.

2. Score it explicitly:

   Item	Points	Scored?
   Column publish	+1
   Column image	+2
   Navigate to column (needs the grasp)	+3
   Row publish	+1
   Row image	+2
   Grasp	+3
   Navigate to bin (needs delivery)	+3
   Place: dropped +2 or gentle +4
   Collisions	-0.5 each

   Remember the two contingencies: navigation to the column scores nothing unless the grasp succeeded, and navigation to the bin scores nothing unless the book was delivered. A run that drove perfectly and failed the grasp scores 6, not 9.

3. Append the row to results/trials.csv. Schema in docs/BLUEPRINT.md §7. Never skip a bad run. Failed trials are the useful data and section 6 of the report is graded on them.

4. Append to docs/BUILD_LOG.md using the standard format, with the rung reached and the actual score.

5. Diagnose failures against the table in .claude/agents/qa.md, and record which hypothesis you're acting on.

6. Record parameter changes: old value, new value, observed effect. If config/params.yaml changed, note which keys and why.

7. Update .claude/PROJECT_NOTES.md only if a build-order gate was genuinely met. A 17 of 20 pass rate does not close a gate that requires 20 of 20.

The five report trials are a special case

Section 5 of the report requires scores across 5 simulation trials. That set has rules:

• Five consecutive trials, each on a fresh world load
• No parameter changes between them. Changing anything mid-set invalidates it as a dataset
• All five logged in full, including bad ones
• Requested column and colour varied across the five

A set with one characterised failure is honest data with a story. A set where the bad runs were quietly re-run is a fabrication, and the limitations section is graded, so the bad run earns more than it costs.

Rules

• Never round up the test rung. "Ran once" is not "works". See .claude/rules/workflow.md.
• One variable at a time. Changing three parameters between runs tells you nothing about which mattered.
• Record failures in full. The failed runs are the useful data; the successful ones just confirm it.
• Break grasp results out by row. Row-dependent failure points at torso positioning; uniform failure points at the grasp pose.
• Don't chase the grasp before the two perception nodes are at 20 of 20. The cheap points come first, per docs/SCORING.md.
• Write plainly: no em dashes, no emoji in headings, no title case.