LoopX Benchmark Workflow
Use this skill to operate or analyze a LoopX-managed benchmark experiment. The builtin
benchmark-toolkit capability owns provider-neutral experiment state and
integrity boundaries. This packaged skill is its task-triggered Agent playbook.
An assigned solver follows its task instructions and current execution contract.
The words “benchmark”, “evaluation”, or “submission” in that task do not grant
the operator role or require experiment-board discovery. Use this workflow when
the requested work actually includes run management, analysis of an active run,
or post-run analysis; an explicit request to use the skill still applies. Keep
the solver's task-local validation and authorized submission path distinct from
experiment management.
The capability is catalog-ready without a per-Goal enable switch. Installing
this skill does not grant runner, shell, network, credential, private-evidence,
or Goal mutation authority. Respect the selected todo's required capabilities,
any external provider binding, host permissions, and user gates.
Capability surface
loopx capability show benchmark-toolkit --format json — catalog entry with
usage hints, role boundaries, and the post-run case-insight template.
loopx benchmark --help — subcommands (experiment-board-show,
experiment-board-upsert, source-revision-fence, integrity-qualification,
classify-artifacts).
Share a study through the public-safe contract
When another benchmark developer needs portable study data, use the capability's
typed study flow rather than sharing a runner-specific ledger or raw evidence:
- Validate
benchmark_study_manifest_v0 with benchmark study-validate.
- Wrap one allowlisted manifest, experiment-board row, redacted insight, or runtime
observation with
benchmark upload-envelope.
- Run
benchmark upload-local without --execute first, then explicitly execute
against a caller-selected local JSONL store.
- Verify the record/digest/revision binding with
benchmark upload-readback.
- Derive the campaign/arm/case/run packet with
benchmark study-dashboard; pass a
compact four-arm contract only when the study preregistered that design.
For case_insight_projection, first upload the same run's active terminal
experiment-board row with insight.status=complete. The case, run, and outcome
must match; the run row remains the only arm, score, countability, integrity, and
treatment-fidelity authority. Reduce private post-run evidence to bounded prose
and public-safe handles or digests before building the envelope.
The local provider is a no-network simulation. It does not grant remote upload,
publication, credentials, retention, or benchmark submission authority. Adapters
keep their native metric names and reduce private post-run evidence before envelope
construction.
Share exploratory behavior findings
When the owner authorizes selected behavioral observations but not a complete
study release, use behavior_finding records and benchmark behavior-report.
See docs/reference/benchmark-behavior-findings.md for the contract. These records
require selection rules, sample denominators, observations, interpretations,
limitations, counterevidence, and evidence digests; they require neither a run-row
upload nor a full study manifest and have no score authority.
Freeze the authorized disclosure projection before rendering. Review the same
scope in visible text, foldouts, embedded data, downloads, and PR attachments.
Permission to share duration does not grant permission to share outcome totals
or deltas. Schema validity and a producer redaction attestation are not publication
approval or verification of unshared evidence. Keep selected-case observations
explicitly exploratory and retain the relevant limitations and counterexamples.
Select the operating lane
- Inspect or explain: use
capability show and benchmark --help; remain
read-only. Do not create an experiment-board row merely because the user asks
what the toolkit does.
- Plan, select, or launch a run: follow the experiment sequence below. The
first action is to read the board; a launch still requires an authorized
runner and admitted source.
- Monitor an active campaign: read the board and runtime-owned projections;
update only on material run transitions. Do not manufacture progress from a
timer tick.
- Diagnose effectiveness or trajectory efficiency: follow the analysis below.
For active runs, use authorized solver/runtime observations and released score
projections; keep findings provisional and hidden evaluator evidence closed.
- Analyze a terminal run: wait until solving is terminal and scoring is
complete before reading hidden evaluator evidence or writing a case insight.
For a generic library microbenchmark or an eval with no LoopX Goal/board, use
the task's normal tools instead of imposing this workflow.
Diagnose effectiveness and trajectory efficiency
Start with the task's real success criterion and evaluator behavior, then explain
which work produced useful progress. Keep score quality and operational efficiency
separate: fewer bytes, calls or tokens are not proof of a better task outcome.
- Read the board and align source, model, task, budget, feedback mode, sampling,
evaluator and concurrency. Compare common elapsed windows and label unmatched
history as diagnostic. Missing or invalid scores remain distinct from zero;
use the task's native ranking when identifying a retained best result.
- Inspect representative early, middle and late trajectory segments, including
stalls and counterexamples. Record the selection rule and sample denominator.
Connect actions to changed artifacts, validation and scored snapshots; count
planning, control reads, tool recovery and actual task work separately. Use
hidden task/evaluator evidence only after solver and scoring are terminal.
- Separate candidate causes: task difficulty or strategy, model token throughput,
control interaction overhead, tool failures, and evaluator/feedback delay.
Align artifact capture, grading and delivery times with solver activity.
Compare both wall time and useful progress per token/active work interval where
measured; a score slope or a token-rate difference alone cannot identify the
cause. Preserve unknowns when telemetry is absent.
- Turn the strongest supported cause into a small discriminating intervention
or ablation within existing authority. Reuse current qualified baselines and
change one relevant factor where possible; disclose unavoidable confounds.
Prioritize expected task benefit and technical depth over packet size or PR
count. If repeated reads are wasteful, verify the consumer's obligations before
reducing them; a planning or recovery change must still lead to useful work.
- Report observations, causal hypotheses and validated effects distinctly,
including regressions and remaining uncertainty. Use
loopx-performance-diagnosis only for
an evidenced owned-process cost; return its measurements to this task-level
analysis. Do not substitute profiler hotspots for outcome evidence or require
profiling to inspect duplicate content and unnecessary interactions.
For recorded model-request totals, use loopx benchmark trajectory-pack-register
with --usage-jsonl and --usage-session-id, then read
solver_usage.projection.request_coverage. Registration does not run an analyzer;
an unfinished final Turn requires the explicit --include-incomplete-usage
option. Live-tail usage_cumulative and token_count observations retain the
native session-counter basis, which may exclude compaction. Only verified
request_coverage.all_requests supplies the reconciled recorded-request total;
retain its counter_basis, capture window and limitations. Compaction is a
subset and its mirrored record is not another request. Do not add either to
the session totals. Missing or unavailable coverage remains unknown. This
source selection does not rewrite Goal accounting or establish billing coverage.
Experiment sequence
Read the experiment board before launching or selecting a case.
loopx benchmark experiment-board-show --goal-id <GOAL_ID> --format json
Inspect baseline, treatment, explore, countability, effort, and insight rows
before choosing the next arm.
Qualify the source revision before each new run admission.
loopx benchmark source-revision-fence \
--source-checkout <clean-source> \
--expected-revision <PIN> \
--observed-reference-revision <OBSERVED_HEAD> \
--require-admitted --format json
The fence fails closed unless the clean pinned source matches the observed
reference head.
Preview, then preregister or mark the run row when it starts.
loopx benchmark experiment-board-upsert --goal-id <GOAL_ID> \
--row-json <running-row.json> --format json
loopx benchmark experiment-board-upsert --goal-id <GOAL_ID> \
--row-json <running-row.json> --execute --format json
The running row uses status=running, empty metrics, and
countability={integrity_qualified:false, official_result_present:false, score_countable:false}. Keep the same stable run_id for every transition.
Upsert returns the normalized affected row and write receipt by default.
Check its identity and preview/write outcome directly. Read the board again
when the next comparison needs it, using the same route and relevant
--benchmark-id/--study-id; do not re-fetch a whole study merely to confirm
one update. --include-board restores the previous full-board response for
consumers that need it. A row receipt does not establish comparison eligibility.
Preview and upsert terminal score, countability, effort, and insight.
First run integrity qualification. An automated restricted-access match is a
countable suspicion, not a cheating verdict. After solver and scoring are
terminal, inspect the real solver trajectory, tool results, and final
workspace. Pass a compact
benchmark_restricted_access_adjudication_v0 only after that review; confirm
cheating only when restricted material was actually disclosed and causally
entered a solving or validation decision.
loopx benchmark experiment-board-upsert --goal-id <GOAL_ID> \
--row-json <terminal-row.json> --execute --format json
The terminal row sets status=completed, fills metrics (primary metric plus
guardrails), and updates countability. Only mark score_countable=true when
integrity_qualified=true and official_result_present=true. Fill effort
and set insight.status to complete after the post-run analysis.
For non-baseline arms, also reduce the reviewed mechanism facts separately:
loopx benchmark treatment-continuation-receipt \
--observation-json <compact-post-run-observation.json> --format json
This receipt distinguishes qualified startup from post-start semantic control
persistence. It is analysis-only and must not change score countability,
integrity qualification, treatment fidelity, or matched-pair eligibility.
Read matched comparisons before selecting the next arm.
loopx benchmark experiment-board-show --goal-id <GOAL_ID> --format json
Only claim paired results from matched_pair_countable comparisons. Keep
diagnostic-only explore rows in a separate evidence lane.
Quota interruption and checkpoint continuation
When an authorized run encounters hard native account exhaustion, preserve its
actual failure and task progress. Verify the exact job, terminal/runtime state,
source account and original budget before selecting a replacement. Unknown
quota, login errors or generic rate limits are not proof of exhaustion. Reuse
the installed native account provider's read-only observations and TypeScript
selection through python -m benchmark.runtime.codex_accounts; inspect the
preview, then execute only within the owner's explicit account allowlist.
Never close the desktop or switch its global login to repair an isolated job.
If the worker was removed, prefer its latest verified complete checkpoint to
a fresh task. Retain the original attempt and source, restore native session
history/research/incumbent/opportunity counters/cooldown through the existing
receiver, and use fresh destination identities. Preregister the continuation
and show its source and inherited results without merging attempt curves.
Confirm actual adopted context, current replan and subsequent task work.
Prepared credentials or a running container alone do not prove recovery.
Follow the EdgeBench account-continuation protocol for the native controller's
paused remaining-duration clock. An unchanged absolute deadline requires an
external finite owner cutoff, including setup and cleanup; no implicit phase
extension, replenished evaluation opportunities or repeated cold launches.
The helper prepares a new private login only. Existing Official sessions need
their own authorized idle handoff; never overwrite an active credential file
using the preparation helper. If no allowed account is freshly usable, retain
the observed reset/backoff and avoid repeated model probes.
Run-row contract
benchmark_id, study_id, case_id, run_id, arm_id, arm_role,
attempt, status, observed_at, model_id, protocol_id,
comparison_protocol_id, claim_scope, primary_metric,
guardrail_metrics, metrics, countability, treatment_fidelity,
effort, insight are the canonical row fields (schema_version =
benchmark_experiment_board_row_v0).
- Baseline rows must use
treatment_fidelity=not_applicable and cannot name a
comparison_anchor_run_id. Non-baseline rows must name a
comparison_anchor_run_id.
- Metrics are
{"name": {"value": <number>, "unit": <str>, "higher_is_better": <bool>}}; at most 16 entries. The primary_metric must
not also be a guardrail metric.
score_countable requires status=completed, integrity_qualified=true,
and official_result_present=true. score=0 is a valid completed result.
Source, integrity, and artifact boundaries
source-revision-fence is read-only and caller-observed: it performs no
fetch, install, or launch. It blocks new admissions only.
integrity-qualification reduces private trajectory and runner isolation
evidence to a compact public-safe receipt (hashes, counts, reason codes).
- Scanner hits for restricted source access, host-boundary escape, or credential
probes set
restricted_access_review=suspected while keeping the run
score-eligible. Use --restricted-access-adjudication-json for the post-run
agent decision; only confirmed disclosure plus causal use disqualifies the
score. An observed credential value remains fail-closed. A broad shell probe
that bypasses an explicit typed task-environment contract is also a hard
policy violation rather than reviewable suspicion.
classify-artifacts classifies benchmark artifact paths without reading them;
use it before reading or publishing any candidate artifact.
- The solver lane must not read hidden tests, verifier sources or gold answers.
During solving, official feedback is limited to what the declared run protocol
releases to that solver. The post-run analyst may read full private evidence
only after the solver is terminal and scoring is complete.
capability bind selects an external provider implementation for a Goal; it
is not the activation mechanism for this builtin capability. Todo
required_capability fields remain runtime prerequisites, not product
capability switches.
Curate live comparison views
When the user wants a persistent experiment overview, keep a stable per-task
entry point backed by an explicit maintained selection, rather than sending a
new long run-query URL after every restart. Keep task switching and full history
one interaction away. Use the existing board/runtime projections for run state
and the provider's authorized score projection; a display selection must not
become a second source of score, integrity, or countability truth.
- Include the requested baseline families and feedback modes, current experiments,
and important mechanism ablations. Do not silently reduce baselines to the
official runner: single-task and native-Goal controls may be essential. If a
requested baseline is unavailable, say so instead of substituting another arm.
- By default, move superseded or problem-stopped attempts out of the core view,
while retaining their original traces, scores, retirement reason and replacement
reference in history. Never select by score or hide a valid low-scoring arm.
An explicitly requested historical baseline may remain visible, with its actual
status and incomplete duration labeled; display inclusion is not qualification.
- Keep historical baselines distinct from current experiments. Expose each arm's
runner setting, source/scorer version, feedback mode, budget and sampling cadence
through concise labels and accessible details. Mark unmatched versions or budgets
as diagnostic context rather than implying a causal comparison.
- Compare common elapsed sampling windows. Show each completed score promptly,
with pending, evaluating, failed and missing points distinguishable; never fill
missing scores with zero or splice different attempts into one curve. Preserve
original capture times and terminal samples.
- Reconcile the selection on admission, replacement and retirement while keeping
the entry URL stable. Maintain campaign-specific identifiers in private operator
state; put only reusable guidance in the shipped skill.
- Verify the rendered view: requested baselines and active ablations appear,
retired attempts are reachable through history, and links survive a selection
update. Review the populated first viewport for readable labels and navigation;
a correct query alone does not establish a usable comparison view.
Campaign monitoring and post-run insight
- When a campaign starts and the caller authorizes ongoing monitoring, add one
continuous_monitor todo. Refresh aggregate score/coverage and write
benchmark_case_insight_v0 on material scored-case transitions, with bounded
periodic reviews while the campaign remains active.
- Treat that monitor as an observation lane, not executable delivery. When a
material poll discovers bounded repository, runner-repair, or experiment work,
use
quota monitor-poll --material-change --next-agent-todo with explicit
--next-action-kind, repository, and required capabilities so it creates an
independent runnable advancement_task. An unchanged poll creates no successor
and spends no delivery quota.
- If the main campaign advancement Todo is waiting for a monitor transition, keep
it
open and pair resume_when=monitor_changed:<monitor-todo-id> with an
already-created independent runnable successor. Do not mark the wait blocked,
and do not treat the monitor itself as delivery work.
- Report only public-safe conclusions (countable baselines, countable
treatments, matched pairs, aggregate primary metric by arm, improved/flat/
regressed pair counts). Never copy raw private evidence into a user update.
- After a solver stops and scoring completes, read the task, real trajectory,
final workspace, hidden tests, verifier, and failure/score details; write one
benchmark_case_insight_v0 explaining the decisive evidence, why the outcome
happened, and what LoopX should test next.
- For treatment arms, record whether qualified startup was followed by semantic
Todo transitions, technical replans, or control closeout. Use
startup_only
only when a complete authorized post-run review observed no such transition;
otherwise absence is unknown. Keep terminal settlement separate.
- Do not send a repetitive user update when nothing material changed.
For fixed-duration studies, use the campaign report's duration policy for every
declared arm, including Official. At each terminal transition reconcile its
active solver intervals and any maintained make-up obligation with the report.
Only waive a short run when its arm is expressly allowed and its own audited Goal
completed normally. Keep unknown duration pending evidence. Keep provider holds
visible with the outstanding duration and the condition for resuming. Preserve
session, scoring opportunities and cooldown when resuming; record fresh runs
separately. A duration shortfall alone never authorizes overriding a stop.
1---2name: loopx-benchmark3description: "Operate or analyze LoopX-managed benchmark experiments through benchmark-toolkit, including trajectory-based effectiveness and efficiency diagnosis, matched comparisons, run selection and integrity. Solving an assigned task under an existing runner does not by itself select this skill. Excludes casual benchmark discussion and ordinary software microbenchmarks."4metadata:5 source: https://skillmd.com/skills/tuyv/loopx-benchmark6---78# LoopX Benchmark Workflow910Use this skill to operate or analyze a LoopX-managed benchmark experiment. The builtin11`benchmark-toolkit` capability owns provider-neutral experiment state and12integrity boundaries. This packaged skill is its task-triggered Agent playbook.1314An assigned solver follows its task instructions and current execution contract.15The words “benchmark”, “evaluation”, or “submission” in that task do not grant16the operator role or require experiment-board discovery. Use this workflow when17the requested work actually includes run management, analysis of an active run,18or post-run analysis; an explicit request to use the skill still applies. Keep19the solver's task-local validation and authorized submission path distinct from20experiment management.2122The capability is catalog-ready without a per-Goal enable switch. Installing23this skill does not grant runner, shell, network, credential, private-evidence,24or Goal mutation authority. Respect the selected todo's required capabilities,25any external provider binding, host permissions, and user gates.2627## Capability surface2829- `loopx capability show benchmark-toolkit --format json` — catalog entry with30 usage hints, role boundaries, and the post-run case-insight template.31- `loopx benchmark --help` — subcommands (experiment-board-show,32 experiment-board-upsert, source-revision-fence, integrity-qualification,33 classify-artifacts).3435## Share a study through the public-safe contract3637When another benchmark developer needs portable study data, use the capability's38typed study flow rather than sharing a runner-specific ledger or raw evidence:39401. Validate `benchmark_study_manifest_v0` with `benchmark study-validate`.412. Wrap one allowlisted manifest, experiment-board row, redacted insight, or runtime42 observation with `benchmark upload-envelope`.433. Run `benchmark upload-local` without `--execute` first, then explicitly execute44 against a caller-selected local JSONL store.454. Verify the record/digest/revision binding with `benchmark upload-readback`.465. Derive the campaign/arm/case/run packet with `benchmark study-dashboard`; pass a47 compact four-arm contract only when the study preregistered that design.4849For `case_insight_projection`, first upload the same run's active terminal50experiment-board row with `insight.status=complete`. The case, run, and outcome51must match; the run row remains the only arm, score, countability, integrity, and52treatment-fidelity authority. Reduce private post-run evidence to bounded prose53and public-safe handles or digests before building the envelope.5455The local provider is a no-network simulation. It does not grant remote upload,56publication, credentials, retention, or benchmark submission authority. Adapters57keep their native metric names and reduce private post-run evidence before envelope58construction.5960## Share exploratory behavior findings6162When the owner authorizes selected behavioral observations but not a complete63study release, use `behavior_finding` records and `benchmark behavior-report`.64See `docs/reference/benchmark-behavior-findings.md` for the contract. These records65require selection rules, sample denominators, observations, interpretations,66limitations, counterevidence, and evidence digests; they require neither a run-row67upload nor a full study manifest and have no score authority.6869Freeze the authorized disclosure projection before rendering. Review the same70scope in visible text, foldouts, embedded data, downloads, and PR attachments.71Permission to share duration does not grant permission to share outcome totals72or deltas. Schema validity and a producer redaction attestation are not publication73approval or verification of unshared evidence. Keep selected-case observations74explicitly exploratory and retain the relevant limitations and counterexamples.7576## Select the operating lane7778- **Inspect or explain:** use `capability show` and `benchmark --help`; remain79 read-only. Do not create an experiment-board row merely because the user asks80 what the toolkit does.81- **Plan, select, or launch a run:** follow the experiment sequence below. The82 first action is to read the board; a launch still requires an authorized83 runner and admitted source.84- **Monitor an active campaign:** read the board and runtime-owned projections;85 update only on material run transitions. Do not manufacture progress from a86 timer tick.87- **Diagnose effectiveness or trajectory efficiency:** follow the analysis below.88 For active runs, use authorized solver/runtime observations and released score89 projections; keep findings provisional and hidden evaluator evidence closed.90- **Analyze a terminal run:** wait until solving is terminal and scoring is91 complete before reading hidden evaluator evidence or writing a case insight.9293For a generic library microbenchmark or an eval with no LoopX Goal/board, use94the task's normal tools instead of imposing this workflow.9596## Diagnose effectiveness and trajectory efficiency9798Start with the task's real success criterion and evaluator behavior, then explain99which work produced useful progress. Keep score quality and operational efficiency100separate: fewer bytes, calls or tokens are not proof of a better task outcome.1011021. Read the board and align source, model, task, budget, feedback mode, sampling,103 evaluator and concurrency. Compare common elapsed windows and label unmatched104 history as diagnostic. Missing or invalid scores remain distinct from zero;105 use the task's native ranking when identifying a retained best result.1062. Inspect representative early, middle and late trajectory segments, including107 stalls and counterexamples. Record the selection rule and sample denominator.108 Connect actions to changed artifacts, validation and scored snapshots; count109 planning, control reads, tool recovery and actual task work separately. Use110 hidden task/evaluator evidence only after solver and scoring are terminal.1113. Separate candidate causes: task difficulty or strategy, model token throughput,112 control interaction overhead, tool failures, and evaluator/feedback delay.113 Align artifact capture, grading and delivery times with solver activity.114 Compare both wall time and useful progress per token/active work interval where115 measured; a score slope or a token-rate difference alone cannot identify the116 cause. Preserve unknowns when telemetry is absent.1174. Turn the strongest supported cause into a small discriminating intervention118 or ablation within existing authority. Reuse current qualified baselines and119 change one relevant factor where possible; disclose unavoidable confounds.120 Prioritize expected task benefit and technical depth over packet size or PR121 count. If repeated reads are wasteful, verify the consumer's obligations before122 reducing them; a planning or recovery change must still lead to useful work.1235. Report observations, causal hypotheses and validated effects distinctly,124 including regressions and remaining uncertainty. Use125 [loopx-performance-diagnosis](../loopx-performance-diagnosis/SKILL.md) only for126 an evidenced owned-process cost; return its measurements to this task-level127 analysis. Do not substitute profiler hotspots for outcome evidence or require128 profiling to inspect duplicate content and unnecessary interactions.129130For recorded model-request totals, use `loopx benchmark trajectory-pack-register`131with `--usage-jsonl` and `--usage-session-id`, then read132`solver_usage.projection.request_coverage`. Registration does not run an analyzer;133an unfinished final Turn requires the explicit `--include-incomplete-usage`134option. Live-tail `usage_cumulative` and `token_count` observations retain the135native session-counter basis, which may exclude compaction. Only verified136`request_coverage.all_requests` supplies the reconciled recorded-request total;137retain its `counter_basis`, capture window and limitations. Compaction is a138subset and its mirrored record is not another request. Do not add either to139the session totals. Missing or unavailable coverage remains unknown. This140source selection does not rewrite Goal accounting or establish billing coverage.141142## Experiment sequence1431441. **Read the experiment board before launching or selecting a case.**145 ```bash146 loopx benchmark experiment-board-show --goal-id <GOAL_ID> --format json147 ```148 Inspect baseline, treatment, explore, countability, effort, and insight rows149 before choosing the next arm.1501512. **Qualify the source revision before each new run admission.**152 ```bash153 loopx benchmark source-revision-fence \154 --source-checkout <clean-source> \155 --expected-revision <PIN> \156 --observed-reference-revision <OBSERVED_HEAD> \157 --require-admitted --format json158 ```159 The fence fails closed unless the clean pinned source matches the observed160 reference head.1611623. **Preview, then preregister or mark the run row when it starts.**163 ```bash164 loopx benchmark experiment-board-upsert --goal-id <GOAL_ID> \165 --row-json <running-row.json> --format json166 loopx benchmark experiment-board-upsert --goal-id <GOAL_ID> \167 --row-json <running-row.json> --execute --format json168 ```169 The running row uses `status=running`, empty `metrics`, and170 `countability={integrity_qualified:false, official_result_present:false,171 score_countable:false}`. Keep the same stable `run_id` for every transition.172 Upsert returns the normalized affected `row` and `write` receipt by default.173 Check its identity and preview/write outcome directly. Read the board again174 when the next comparison needs it, using the same route and relevant175 `--benchmark-id`/`--study-id`; do not re-fetch a whole study merely to confirm176 one update. `--include-board` restores the previous full-board response for177 consumers that need it. A row receipt does not establish comparison eligibility.1781794. **Preview and upsert terminal score, countability, effort, and insight.**180 First run integrity qualification. An automated restricted-access match is a181 countable suspicion, not a cheating verdict. After solver and scoring are182 terminal, inspect the real solver trajectory, tool results, and final183 workspace. Pass a compact184 `benchmark_restricted_access_adjudication_v0` only after that review; confirm185 cheating only when restricted material was actually disclosed and causally186 entered a solving or validation decision.187188 ```bash189 loopx benchmark experiment-board-upsert --goal-id <GOAL_ID> \190 --row-json <terminal-row.json> --execute --format json191 ```192 The terminal row sets `status=completed`, fills `metrics` (primary metric plus193 guardrails), and updates `countability`. Only mark `score_countable=true` when194 `integrity_qualified=true` and `official_result_present=true`. Fill `effort`195 and set `insight.status` to `complete` after the post-run analysis.196197 For non-baseline arms, also reduce the reviewed mechanism facts separately:198 ```bash199 loopx benchmark treatment-continuation-receipt \200 --observation-json <compact-post-run-observation.json> --format json201 ```202 This receipt distinguishes qualified startup from post-start semantic control203 persistence. It is analysis-only and must not change score countability,204 integrity qualification, treatment fidelity, or matched-pair eligibility.2052065. **Read matched comparisons before selecting the next arm.**207 ```bash208 loopx benchmark experiment-board-show --goal-id <GOAL_ID> --format json209 ```210 Only claim paired results from `matched_pair_countable` comparisons. Keep211 diagnostic-only explore rows in a separate evidence lane.212213## Quota interruption and checkpoint continuation214215When an authorized run encounters hard native account exhaustion, preserve its216actual failure and task progress. Verify the exact job, terminal/runtime state,217source account and original budget before selecting a replacement. Unknown218quota, login errors or generic rate limits are not proof of exhaustion. Reuse219the installed native account provider's read-only observations and TypeScript220selection through `python -m benchmark.runtime.codex_accounts`; inspect the221preview, then execute only within the owner's explicit account allowlist.222Never close the desktop or switch its global login to repair an isolated job.223224If the worker was removed, prefer its latest verified complete checkpoint to225a fresh task. Retain the original attempt and source, restore native session226history/research/incumbent/opportunity counters/cooldown through the existing227receiver, and use fresh destination identities. Preregister the continuation228and show its source and inherited results without merging attempt curves.229Confirm actual adopted context, current replan and subsequent task work.230Prepared credentials or a running container alone do not prove recovery.231232Follow the EdgeBench account-continuation protocol for the native controller's233paused remaining-duration clock. An unchanged absolute deadline requires an234external finite owner cutoff, including setup and cleanup; no implicit phase235extension, replenished evaluation opportunities or repeated cold launches.236The helper prepares a new private login only. Existing Official sessions need237their own authorized idle handoff; never overwrite an active credential file238using the preparation helper. If no allowed account is freshly usable, retain239the observed reset/backoff and avoid repeated model probes.240241## Run-row contract242243- `benchmark_id`, `study_id`, `case_id`, `run_id`, `arm_id`, `arm_role`,244 `attempt`, `status`, `observed_at`, `model_id`, `protocol_id`,245 `comparison_protocol_id`, `claim_scope`, `primary_metric`,246 `guardrail_metrics`, `metrics`, `countability`, `treatment_fidelity`,247 `effort`, `insight` are the canonical row fields (`schema_version` =248 `benchmark_experiment_board_row_v0`).249- Baseline rows must use `treatment_fidelity=not_applicable` and cannot name a250 `comparison_anchor_run_id`. Non-baseline rows must name a251 `comparison_anchor_run_id`.252- Metrics are `{"name": {"value": <number>, "unit": <str>,253 "higher_is_better": <bool>}}`; at most 16 entries. The `primary_metric` must254 not also be a guardrail metric.255- `score_countable` requires `status=completed`, `integrity_qualified=true`,256 and `official_result_present=true`. `score=0` is a valid completed result.257258## Source, integrity, and artifact boundaries259260- `source-revision-fence` is read-only and caller-observed: it performs no261 fetch, install, or launch. It blocks new admissions only.262- `integrity-qualification` reduces private trajectory and runner isolation263 evidence to a compact public-safe receipt (hashes, counts, reason codes).264- Scanner hits for restricted source access, host-boundary escape, or credential265 probes set `restricted_access_review=suspected` while keeping the run266 score-eligible. Use `--restricted-access-adjudication-json` for the post-run267 agent decision; only confirmed disclosure plus causal use disqualifies the268 score. An observed credential value remains fail-closed. A broad shell probe269 that bypasses an explicit typed task-environment contract is also a hard270 policy violation rather than reviewable suspicion.271- `classify-artifacts` classifies benchmark artifact paths without reading them;272 use it before reading or publishing any candidate artifact.273- The solver lane must not read hidden tests, verifier sources or gold answers.274 During solving, official feedback is limited to what the declared run protocol275 releases to that solver. The post-run analyst may read full private evidence276 only after the solver is terminal and scoring is complete.277- `capability bind` selects an external provider implementation for a Goal; it278 is not the activation mechanism for this builtin capability. Todo279 `required_capability` fields remain runtime prerequisites, not product280 capability switches.281282## Curate live comparison views283284When the user wants a persistent experiment overview, keep a stable per-task285entry point backed by an explicit maintained selection, rather than sending a286new long run-query URL after every restart. Keep task switching and full history287one interaction away. Use the existing board/runtime projections for run state288and the provider's authorized score projection; a display selection must not289become a second source of score, integrity, or countability truth.290291- Include the requested baseline families and feedback modes, current experiments,292 and important mechanism ablations. Do not silently reduce baselines to the293 official runner: single-task and native-Goal controls may be essential. If a294 requested baseline is unavailable, say so instead of substituting another arm.295- By default, move superseded or problem-stopped attempts out of the core view,296 while retaining their original traces, scores, retirement reason and replacement297 reference in history. Never select by score or hide a valid low-scoring arm.298 An explicitly requested historical baseline may remain visible, with its actual299 status and incomplete duration labeled; display inclusion is not qualification.300- Keep historical baselines distinct from current experiments. Expose each arm's301 runner setting, source/scorer version, feedback mode, budget and sampling cadence302 through concise labels and accessible details. Mark unmatched versions or budgets303 as diagnostic context rather than implying a causal comparison.304- Compare common elapsed sampling windows. Show each completed score promptly,305 with pending, evaluating, failed and missing points distinguishable; never fill306 missing scores with zero or splice different attempts into one curve. Preserve307 original capture times and terminal samples.308- Reconcile the selection on admission, replacement and retirement while keeping309 the entry URL stable. Maintain campaign-specific identifiers in private operator310 state; put only reusable guidance in the shipped skill.311- Verify the rendered view: requested baselines and active ablations appear,312 retired attempts are reachable through history, and links survive a selection313 update. Review the populated first viewport for readable labels and navigation;314 a correct query alone does not establish a usable comparison view.315316## Campaign monitoring and post-run insight317318- When a campaign starts and the caller authorizes ongoing monitoring, add one319 `continuous_monitor` todo. Refresh aggregate score/coverage and write320 `benchmark_case_insight_v0` on material scored-case transitions, with bounded321 periodic reviews while the campaign remains active.322- Treat that monitor as an observation lane, not executable delivery. When a323 material poll discovers bounded repository, runner-repair, or experiment work,324 use `quota monitor-poll --material-change --next-agent-todo` with explicit325 `--next-action-kind`, repository, and required capabilities so it creates an326 independent runnable `advancement_task`. An unchanged poll creates no successor327 and spends no delivery quota.328- If the main campaign advancement Todo is waiting for a monitor transition, keep329 it `open` and pair `resume_when=monitor_changed:<monitor-todo-id>` with an330 already-created independent runnable successor. Do not mark the wait `blocked`,331 and do not treat the monitor itself as delivery work.332- Report only public-safe conclusions (countable baselines, countable333 treatments, matched pairs, aggregate primary metric by arm, improved/flat/334 regressed pair counts). Never copy raw private evidence into a user update.335- After a solver stops and scoring completes, read the task, real trajectory,336 final workspace, hidden tests, verifier, and failure/score details; write one337 `benchmark_case_insight_v0` explaining the decisive evidence, why the outcome338 happened, and what LoopX should test next.339- For treatment arms, record whether qualified startup was followed by semantic340 Todo transitions, technical replans, or control closeout. Use `startup_only`341 only when a complete authorized post-run review observed no such transition;342 otherwise absence is `unknown`. Keep terminal settlement separate.343- Do not send a repetitive user update when nothing material changed.344345For fixed-duration studies, use the campaign report's duration policy for **every346declared arm**, including Official. At each terminal transition reconcile its347active solver intervals and any maintained make-up obligation with the report.348Only waive a short run when its arm is expressly allowed and its own audited Goal349completed normally. Keep unknown duration pending evidence. Keep provider holds350visible with the outstanding duration and the condition for resuming. Preserve351session, scoring opportunities and cooldown when resuming; record fresh runs352separately. A duration shortfall alone never authorizes overriding a stop.