Trace Harness Launch Failure

Trace Harness Launch Failure — empty sessions, scores from deliverables

Problem

Four training traces contain ONLY the initial task prompt: 1 message, 0 tool calls, 0 API calls, ~1 s duration. No agent reasoning, tool call, or final answer anywhere — yet meta.json carries scores (0.0 / 1.0).

Root cause

Every recorded run died at launch: runs/<task>/stdout.txt shows "HTTP 400: default is not a valid model ID" — the harness sent an invalid model ID instead of the configured model, so the agent never executed. The scores were produced by grading the on-disk deliverables against the exact-match graders in tasks.json — not from any recorded agent behavior.

Fix

• Before analyzing "agent behavior", verify the session actually ran: check api_call_count / tool_call_count / message_count in the trace and stdout.txt for launch errors. Empty trace == no behavioral evidence.
• When traces are empty, reconstruct ground truth by diffing the on-disk deliverable against the task's grader expectation (tasks.json → grader.expected), and read spec.md + input data to explain the score.
• Treat prior-iteration behavioral claims as unverifiable until new traces confirm them; flag evidence status in the run log.
• Surface the harness bug so the runner is fixed before the next iteration, otherwise every proposal rests on inferred data.