Open-Benchmark Methodology
The benchmark measures the Vibe-IC product path, not a benchmark-tuned agent. Every runnable RTL evaluation uses the same solve core as an ordinary IC-design request. Dataset differences are allowed only at the input/output boundary.
Rule 0 — one product entry
For the open RTL suites in benchmark/BENCHMARK_REGISTRY.json, the only
authoring path is:
benchmark_dispatch.py <bench> --solve
-> benchmark_io_adapter.stage (input translation only)
-> task_nature_route (input-only advisory proposal)
-> needs_ai_routing.jsonl (AI judges task nature before runner)
-> benchmark_dispatch.py <bench> --resume (hash-bound AI route validation)
-> vibe_ic_one_shot_runner
--entry-step <Router entry> --exit-step <Router exit>
--execution-mode default (normal product runner)
-> Program candidate or declared AI backup
-> independent hash-bound AI review
-> benchmark_dispatch.py <bench> --resume
-> accepted candidate
-> benchmark_dispatch.py <bench> --score
-> official host scorer
For whole-IC benchmark_clean, use /vibe-ic-all through
vibe_ic_one_shot_runner.py, then use benchmark-verify. The public entry
always executes flow_engine.py against flow/steps.json; the Router controls
its evidence window. A benchmark name never selects a solver. Report canonical
YAML positions and step names; historical IDs remain execution details.
Forbidden alternatives include:
- benchmark-name or problem-id routers, task loops, tier pipelines, solve pipelines, Phase-1 wrappers, prompt exporters, or free-hand authoring loops;
- choosing a route from a CVDP
cid, benchmark label, problem name, expected answer, prior score, or hidden harness; - writing scoreable RTL directly and then calling a gate/scorer;
- publishing a partial previous-failure run as a benchmark result.
benchmark/benchmark_entry_surface_check.py enforces the shipped entry surface
and is called by --solve. If it fails, stop and repair the product entry before
running a benchmark.
GENERAL-CORE / THIN-ADAPTER
Benchmark-specific code may only translate formats or invoke the official scorer. A thin adapter may:
- stage the current record's visible prompt and supplied context into a normal project;
- collect an already-produced candidate;
- package exact accepted bytes into the scorer's required envelope;
- run image/tool preflight and translate official verdict files.
It may not classify the task, author or repair RTL, choose a product step, infer hidden interface facts, assign PASS/FAIL, or use reference values. Logic that works from ordinary prose/RTL belongs in a neutrally named general program and must be reachable from a non-benchmark Phase-1/project flow.
For CVDP, output.context values are hidden golden data. At the post-acceptance
host packaging boundary only, the adapter may read its path keys to construct
the official multi-file response envelope. It must never expose values to the
solver or reviewer and must refuse ambiguous file/module mappings.
Program First, AI Backup, AI authority
This is a sequence, not two independent racing solvers:
- AI confirms or overrides the input-only task routing before any design runner executes. Routing does not author RTL or replace a Program gate.
- Program runs first for RTL through the normal runner.
- If Program emits a candidate, an independent AI reviews the exact frozen RTL hash against the visible prompt/context.
- If Program declares a supported WAIVE or route-level AI backup, AI may author
the missing candidate into the runner-owned RTL directory;
--resumere-enters the same product gates. - If AI agrees with Program, record acceptance for that exact hash.
When Program lacks PASS functional evidence, the AI's current challenge plus
every active inherited challenge must cover every block-eligible structural
item emitted in
program_review_obligations;spec_coverage_check.pymeasures this, and an uncovered item blocks acceptance. - If AI disagrees, it must identify the prompt requirement and provide a prompt-derived executable test that fails on the frozen Program candidate. Prose disagreement alone cannot replace the candidate.
- A repair must pass that immutable challenge, the same runner gates, and a fresh independent AI review. AI is the final semantic authority only after this evidence chain is complete.
Reject only for explicit visible-input obligations; quote the clause imposing each rejected check. General defaults guide authors; reviewer preferences add no unstated latency, reset, full-threshold or clock-phase duties. If several readings satisfy the explicit contract, record an engineering interpretation without rejecting other compatible choices for extra restrictions. Positive examples alone do not imply iff; test clocks and stimulus must obey input assumptions. Rigorously verify every explicit duty and required proof. Compatible choices do not justify blanket acceptance, skipped validation or human clarification.
A structural-only rejection still requires the frozen candidate to PASS the
prompt-derived functional challenge. Supplied-RTL tasks retain the existing
source-bound structural contract and Program functional PASS requirement. A
source-verified spec_generation or completion task using original input may instead declare
vibeic.prompt_structural_contract.v1: name its exact relative candidate_path,
bind the candidate hash and a unique exact prompt quote, and use the task's
immutable native image. Every active functional challenge must PASS and cover
all blocking obligations before native STRUCTURAL_CONTRACT_VIOLATION permits
repair. The repair record carries vibeic.prompt_structural_completion.v1 and
the unchanged structural declaration, rebinding only its candidate hash; normal
completion replays it before a fresh independent review. Only structures
retained at RTLIL_AFTER_PROC_OPT are supported. Functional equivalence cannot
prove source syntax such as assign versus always, and an unsupported abstraction
remains NOT_MEASURED.
No public RTL files does not mean no supplied RTL: the authoritative original
prompt may contain it. Source facts distinguish absent RTL from a single
complete header with no body. The latter requires prompt_header_preservation
in the structural payload, of schema vibeic.prompt_header_preservation.v1,
with prompt_sha256, module, exact original header and header_sha256.
The candidate must preserve the header's significant tokens; only whitespace
and comments may vary. The same binding survives native repair replay. This
narrow structural exception does not allow a functional-only repair to omit its
source contract. Embedded bodies, multiple modules, unsupported source context,
incomplete extraction and malformed public manifests never become text-only.
A functional rejection on a header-only input may use the distinct repair
wrapper vibeic.prompt_header_completion.v1. It binds prompt_sha256,
parent_rtl_sha256, whole-bundle candidate_sha256, challenge_sha256, the
same attributed author and oracle_accessed: false, plus an exact relative
candidate_path, its byte hash candidate_file_sha256, and the unchanged
prompt_header_preservation. Normal completion replays that header proof;
ordinary native gates, the immutable functional challenge and fresh hash-bound
AI review remain mandatory. Header fidelity alone is not functional PASS.
An active structural contract must keep its original structural wrapper and
cannot be replaced by HeaderCompletion. Supplied bodies, multiple/ambiguous
modules and unverified source authority retain their strict source obligations.
Every candidate therefore has two required checks: deterministic product gates
and semantic AI review. program_first_ai_acceptance.json must be COMPLETE
before scoring.
A required Step 4 needs its reference model
When a route's evidence window includes Step 4 and Program cannot derive the
reference, the required sim gate fails on the unfilled hook, and an independent
semantic PASS does not discharge that failure. The AI backup task names the
model in reference_model_handoff; an independent IC expert derives it from the
visible input and fills it before --resume. A model filled or replaced later is
measured by the next --resume on the same RTL bytes, followed by a fresh review.
Editing the RTL is not a way back to the runner without a proven failure.
Autonomous evaluation: IC-expert adjudication of ambiguity
Read the entire visible input, including tables, diagrams, interface declarations
and timing examples. A behavior-affecting source ambiguity must name two
source-bound, mutually exclusive observable results for the same allowed history
or parameter point on which this candidate depends. Use the existing
vibeic.spec_clarification.v1 request with exactly two readings, each carrying
source_sha256, an exact excerpt and the behavior it supports. A private
encoding, missing X/Z policy, unsupported parameter or convention is insufficient.
Later explicit reset or direction requirements can resolve earlier vague prose.
Do not invent a finding from a module name or suspected hidden answer.
When Program authoring detects that finding, record it before deterministic
emission in reports/spec_clarification_review.json and take the real existing
WAIVE. When independent review detects it, use NEEDS_CLARIFICATION without a
fabricated failing test. Both paths go to the existing AI_BACKUP author. Review
handoff preserves the original Program result, candidate, review and tests; it
does not turn Program PASS into a fake WAIVE or AI_REPAIR. Only the coordinator's
mutable RTL workcopy retires so a named AI can freshly author the chosen reading.
The actual AI author reads the complete input, records selected_readings as
one 0 or 1 per request plus rationale and source evidence, writes the RTL and
signs the normal completion. Identical RTL bytes are permitted after fresh
authoring; relabeling retained Program files is not. Native gates and fresh
independent review still decide acceptance. The reviewer assesses compatibility
of the chosen reading and its implementation; the same original ambiguity does
not requeue a completed choice. Immutable failed challenges remain obligations,
with only the existing source-bound supersession path available. A Program's
positive confirmation stays in prior context and cannot lock the AI to its
original choice. Hidden references, testbenches and scores provide no authority.
An erroneous finding may still use the ordinary review-correction path. Never
rewrite the frozen input or count the handoff itself as acceptance.
why_not_bucket_a: deciding whether contextual hardware requirements support a defensible interpretation of conflicting clauses requires IC design judgment; no generic regex, clause vote or source-type priority can decide it. The deterministic handoff and immutable review boundary remain program-owned.
Correcting an immutable review question
An incorrect source-bound review question must not be overwritten, and correct
RTL must not be changed just to obtain a new review key. The normal coordinator
may apply --resume --review-correction <REQUEST> to an unaccepted, unpublished
task. Its vibeic.benchmark.ai_review_correction.v1 request binds the current
task digest, prompt, candidate, review and challenge hashes, an attributed blind
AI author, and a rationale of at least 80 characters. This is a fresh-review
operation, not an acceptance, repair permit or challenge supersession.
A NEEDS_CLARIFICATION review without verification_test is eligible only if
the ordinary review validator confirms a valid source question. The request must
include challenge_sha256 explicitly: the exact supplemental file hash when the
task's challenge path exists, or null when absent. The archive records existing
uncontracted diagnostics as PRESERVED_UNCONTRACTED_SUPPLEMENTAL_NOT_PROMOTED,
and absence as ABSENT_NO_VERIFICATION_OBLIGATION. Neither invents an executable
contract the prior question did not claim. Every already-active inherited
challenge remains unchanged and binding; an existing verification_test keeps
its ordinary validation and inheritance rules. Stale material, invalid questions
and file-presence changes are refused. Original evidence remains immutable and
the new review/test paths must receive fresh independent evidence.
When a correction proposes a typed witness, its rationale does not attach it.
The fresh independent reviewer must assess the entire witness against the full
visible source and bind only a source-qualified witness as the current
verification_test, with its exact hash and source evidence, at the actual newly
issued task.challenge_path; preview paths provide no authority. Preserve and
execute every active inherited challenge and require ordinary native proof and
combined coverage. Reusing the prior current test does not establish the proposed
witness's binding; file presence does not qualify an uncontracted diagnostic.
why_not_bucket_a: benchmark_dispatch.py enforces hashes, paths and inheritance;
selecting a semantically valid witness requires IC-expert judgment.
Blind clean-room evaluation
A canonical run is a fresh full-dataset run. Do not read or inherit:
- prior samples, run artifacts, result files, or agent memory;
- sibling problems' solutions, testbenches, build scripts, or conventions;
- hidden harnesses, reference/golden outputs, or the host scorer during solve, review, repair, or optimization;
- prior failing IDs as a way to narrow the evaluation.
Codex launchers that inject workspace memory must disable memories for blind
actors, for example codex exec --disable memories --disable multi_agent.
Verify the actual native developer context; fork_turns=none alone does not
establish a clean room. An injected historical memory summary is disqualifying.
Preserve native per-content provenance in the unfiltered export. A text banner
alone cannot establish which provider supplied that context.
--solve owns run-directory creation, writes clean-room metadata, pre-creates
the transcript directory, and processes every current dataset case. Separate
scaffolding, benchmark-local authoring, and partial prior-failure evaluation are
not entry points. --limit is diagnostic only and its run is ineligible for
canonical scoring.
Export author/reviewer transcripts to <run>/transcripts/. The score front
door runs benchmark_clean_room_check.py and blindness_audit.py. Missing
transcripts require an explicit disclosure; a detected oracle access blocks
the score.
Batch-dispatch ORCHESTRATION RULES — Shapes B/C (ORGANIC-20260605,
REQUIRED): these bind every shape whose AI worklists fan out over more than
one agent — Shape C and Shape B alike; Shape D is a single project with no
fan-out and is exempt. Batch granularity (one agent per worklist slice, never
per problem), disk truth (progress is read from the run directory, never
from agent returns; resume = --resume), transcript export by default, and
the rate-limit resilience ladder (1-agent CANARY, then 2–4 concurrent with
completion-driven dispatch) are stated in
benchmark/blind_instructions_shape_c.md § ORCHESTRATION RULES and bound
into Shape B by benchmark/blind_instructions_shape_b.md.
Blind evaluation is the exam. Do not loop against official verdicts. Post-score analysis may diagnose and capture a general enhancement, but the resulting plugin change is measured only in a new clean-room run.
Canonical commands
python3 programs/benchmark_dispatch.py <bench> \
--solve --dataset <DATASET> --run <FRESH_RUN>
# Complete needs_ai_routing.jsonl from the visible input only. Then --resume
# validates ALL issued route records and starts the normal Program runner.
# If even one route is missing, invalid, or needs clarification, zero design
# runners start; correct only that response and repeat --resume without
# restaging inputs or re-authoring completed route responses:
python3 programs/benchmark_dispatch.py <bench> \
--resume --dataset <DATASET> --run <FRESH_RUN>
# Complete only the resulting runner-declared AI backup/review/repair worklists:
python3 programs/benchmark_dispatch.py <bench> \
--resume --dataset <DATASET> --run <FRESH_RUN>
# Repeat --resume until program_first_ai_acceptance.json is COMPLETE.
python3 programs/benchmark_dispatch.py <bench> \
--score --dataset <DATASET> --run <FRESH_RUN>
CVDP scoring additionally requires --scorer-root <official-cvdp-root> (or
CVDP_BENCHMARK_ROOT) and the exact official simulation images. Run the shipped
EDA image preflight before scoring. Do not substitute tools silently.
The Program re-entry: re-running a FIXED Program on a preserved signed input
When a Program gate transformed an AI repair into different bytes, the existing
AI_REPAIR_FINAL_PROVENANCE_REQUIRED refusal remains authoritative. An upgraded
Program can be re-entered against the preserved, originally signed input with
the ONE Program re-entry operation:
python3 programs/benchmark_dispatch.py <bench> --resume \
--dataset <DATASET> --run <RUN> --worker-threads 1 \
--program-regate <RUN>/program_regate_request.json
An unaccepted AI_BACKUP candidate may use the same operation after a general
Program fix. Re-enter its existing frozen RTL through native validation, retain
its backup author record and AI_BACKUP attribution, and require fresh independent
review. Re-entry does not grant new authoring or immediate acceptance.
--program-retry is a DEPRECATED alias of --program-regate: it runs the same
merged operation and prints a deprecation line on stderr. That line names the
removal window, and it reads the version from the plugin's own manifest rather
than a literal typed here — the merge's landing version is assigned at landing,
so any number written into this page would be a guess that the plugin itself
could contradict. Run --program-retry, or benchmark_dispatch.py --help, to
see which version applies to the plugin you are actually running. Giving both
names in one resume is refused, not ordered.
Before the merge these were TWO operations (--program-regate, landed v1.17.63,
and --program-retry, landed v1.17.71) for one job; issue #2047 merged them.
Those two numbers are history — where each operation actually landed — and stay
written down; the removal window is a forward claim and is not.
BOTH identities, always. The merged operation refuses unless the Program moved in BOTH senses, because neither is necessary or sufficient for the other: a real fix can land with no version bump, and a version can move with the executable sources untouched.
- the declared VERSION pair —
program_version_beforemust be the version that produced the preserved input,program_version_aftermust be the running Program, and the two must differ (an unchanged version is a loop, not a fix); - the executable SOURCE TREE —
program_identity, which fingerprints installed runtime source and configuration including dirty edits, must equal the running fingerprint, and must differ from the identity of any prior re-entry on this task. It is checked again after the runner.
Unaccepted PROGRAM candidates with a valid current review use common request fields after an upgrade, without signed-input fields. Ordinary workers regenerate unchanged public input, retain prior projects and challenges, and require fresh AI review.
The following signed-input rules apply only to an unaccepted, unpublished
AI_REPAIR task with a valid signed input snapshot. It does not authorize AI
edits, refresh a signature, supersede a test, or accept a result. The normal
repair boundary automatically saves both repair_input_candidate_snapshot and
the pre_gate_input manifest before running gates; the request must name a
candidate manifest whose exact RTL hash matches the unchanged signed repair
record and whose source_rtl_paths exactly match the task's working RTL paths,
and the task's own pre-gate manifest must describe the same signed bytes.
Reconstructing an unsigned input, supplying arbitrary RTL files, and other
candidate origins are unsupported and refused.
The request is a JSON object with these fields:
| Field | Required binding |
|---|---|
schema |
vibeic.benchmark.program_regate.v1 (the legacy ...program_retry.v1 is still accepted) |
id |
Exact pending task ID |
task_sha256 |
SHA-256 of json.dumps(task, ensure_ascii=False, sort_keys=True) |
prompt_sha256 |
Task's current prompt hash |
stale_output_sha256 |
Task's frozen output hash (legacy spelling rtl_sha256 also accepted) |
signed_input_sha256 |
Preserved input hash, equal to signed repaired_rtl_sha256 (legacy spelling input_rtl_sha256 also accepted) |
repair_record_sha256 |
SHA-256 of the unchanged signed record's exact file bytes |
input_manifest_path, input_manifest_sha256 |
Preserved input manifest and its exact file hash |
program_version_before, program_version_after |
The declared version pair; _after must be the running Program |
program_identity |
Current benchmark_dispatch._program_source_identity() object |
author, blind |
Attributed blind AI author (kind: "AI", a named model, oracle_accessed: false) |
rationale |
Explanation of the re-entry, at least 80 characters (legacy spelling reason also accepted) |
review_sha256, challenge_sha256 |
Exact current proof file hashes if a current review/test exists; otherwise omit or use null |
Where the two front doors spelled one field differently, both spellings are accepted so no caller breaks; supplying both under CONFLICTING values is refused, because an ambiguous identity is not an identity.
What it refuses. A wrong request or task schema; a missing or duplicate task
id; a candidate outside PROGRAM/AI_REPAIR; a stale task_sha256, prompt_sha256,
stale_output_sha256, repair record, input manifest or Program identity; a
request naming no source identity at all; an absent, too-short or unattributed
rationale or author; an unchanged, unnamed or non-running version pair; a signed
input that is not the hash the author signed, or that the gates never changed; a
preserved input that is missing, drifted, internally inconsistent, made by a
different Program version, or that does not cover the working RTL file set; a
work tree that drifted from the frozen gate output (a hand edit cannot be
smuggled through a re-entry); prompt or Phase-1 provenance drift; a task already
accepted in either solve_report.json or the acceptance report, or already
published; a non-canonical response path or a project that is not the
runner-owned one; a missing, drifted or non-inherited challenge; a task that also
has a pending AI backup; an occupied archive or fresh review/test path; any
source changing during preparation; and a request, snapshot or evidence file that
is not a regular file inside the run, including any symlink traversal. Normal
relative project links to locations inside the same project are preserved,
including missing optional targets; absolute or escaping project links are
refused.
Under the existing coordinator lock, the operation runs the ordinary supplied-RTL
validation entry (2) with the original declared exit, in a staged project.
Collection uses the normal runner predicate; the exact return code and raw
reports remain available. It preserves the complete prior project, prior task,
solve/acceptance/worklist bytes, and any current review/test under
program_regates/<id>/<request-sha>/. Staged gate outputs remain at their
original paths so references in reports remain usable. Current and inherited
challenges remain obligations of the fresh independent review. Program actions
are recorded separately from the original AI repair author: the transition
records attributed_to: "PROGRAM", author_signature_unchanged: true and
repair_authorized: false, and the author's signature is bound to the preserved
signed INPUT rather than to the Program's output.
A successful re-entry freezes a candidate in a fresh request-addressed snapshot,
creates unoccupied review/test paths, and returns 2 with PENDING. Complete
the fresh independent review and invoke ordinary --resume afterward. If the new
output still differs from the signed input, the unchanged final-provenance check
still refuses acceptance. Existing signatures and historical challenges cannot be
automatically rebound to the new output.
Runner failures preserve the original project/task and write a bound failure
record. An interrupted operation without a valid terminal record emits
PROGRAM_REGATE_REFUSED and blocks ordinary resume. A replayed request over a
journalled re-entry is REFUSED, never reported as already-applied. Reconcile the
immutable intent, staged output, preserved prior project, and transition before
continuing; automatic rollback and guessed recovery are deliberately unsupported.
Do not delete a journal or overwrite owner edits to make resume advance.
Before invoking any official host scorer, run the deterministic cwd guard over the exact design and scorer working directory (and testbench when applicable):
Run the existing official scorer from its documented working directory and preserve actor isolation. A separate working-directory receipt is not a scoring result.
A non-zero guard verdict blocks scoring; do not treat it as advisory.
Four-stage attribution
Record every problem, including failures, in a per-problem table:
- Routing — AI semantic decision plus the prompt/context evidence, selected nature, general entry step, evidence class, and exit step.
- Solving — Program, declared AI backup, or evidence-backed AI repair; name the emitted candidate hash and reason for any handoff.
- Verifying — deterministic runner gates and independent AI review; neither substitutes for the other.
- Looping — no loop, AI backup, AI challenge/repair/re-review, optimization/PPA, or unresolved tool/dataset defect. State who acted and what evidence closed the loop.
The aggregate counts must reconcile exactly to the dataset denominator.
Module-name conflicts when prose and directory disagree (#482)
When the supplied prompt's module name and a supplied directory / file leaf DISAGREE, compare their declared roles against the whole INPUT: the interface header, instantiations and any explicit file-layout convention. A directory leaf is not automatically the required module name because a hidden TB might use it. Record the conflict and the input-supported interpretation in the sample header.
Close-loop and re-author actors preserve the source-defined interface. Neither a prior PASS nor a FAIL establishes which supplied name is a typo. Do not rename a module merely to reproduce a scorer outcome. If the complete INPUT does not settle the conflict, send it to AI authoring with that ambiguity named.
why_not_bucket_a: distinguishing a typo from an intended wrapper name needs contextual judgment about the supplied interface and file layout; no generic regex or hidden naming convention resolves it. Keep the selected source name consistent through emission and verification.
Official scoring and tool disclosure
Scoring is a host-only post-generation step. Use the benchmark's official testbench, image, command, cwd, module naming, and response format. Report:
- dataset identity/hash and exact denominator;
- plugin version/commit, MCP/EDA image digest and tool versions;
- official scorer command and substitutions (or
none); - raw PASS/FAIL/NOT_MEASURED counts and official logs;
- Program/AI routing, solving, verifying, and looping totals.
Never rewrite the official score to compensate for suspected bad goldens. Keep the official number and separately disclose evidence-backed dataset defects.
Failure triage
For every non-PASS, separate:
- product defect: Program/AI/gate/adapter behavior is wrong;
- tool gap: required official capability is absent or incompatible;
- incomplete/ambiguous public specification;
- suspected defective dataset/golden;
- host failure or NOT_MEASURED.
A dataset-defect or floor claim needs a prompt quote, official failure evidence, and a control showing the prompt-correct interpretation cannot satisfy the official oracle. A known-systematic tool blocker must cite its tracking issue number. Never infer a floor merely because one attempt failed.
After writing the per-problem triage record, validate both its internal consistency and whether every general recovery was absorbed into the product:
Retain the failed case, native evidence, diagnosed cause, fix location and verification scope in the benchmark report. An unresolved case remains visible in the result population.
Review every triage entry against its actual failed input, tool output and implemented fix. Do not let a record schema or status label replace that evidence.
Either non-zero verdict blocks the result handoff.
Category D — a tool-substitution gap is FORK-FIXABLE, not a floor
When the official testbench uses a simulator-only construct, or a language
feature our open-source substitute cannot yet run, that is FORK-FIXABLE — not
a terminal floor. We fork the EDA tools and ship them as vibeic-eda, so
"our tool cannot do X" is an engineering backlog item against the fork, routed
to tools/vibeic-eda/FIX_STATUS.md.
The mechanical half of the detection is a program, not a judgement: programs/ tb_vcs_only_construct_detect.py scans the failing testbench for the
iverilog-rejecting VCS/Xcelium-only constructs — array-aggregate '{...} init,
break; / continue;, std::randomize, $urandom_range, unique/priority case, join_none, queue ops — and reports the offending line, the
FORK-FIXABLE disposition and the FIX_STATUS.md route. Use it to
auto-classify D; A/B/C and E-H stay judgement. This is scoring-side syntax
inspection: keep hidden harness contents outside author/reviewer environments.
Only a general capability diagnosis, not expected design behavior, returns to
input-derived recovery.
Before labelling a residual Category D, use the FLOOR-proof controls below. An authorized host-side scorer may build and run its unmodified reference under a tool that supports the feature. That result diagnoses the scoring/tool setup; it supplies no authoring requirement or permission to inspect golden RTL.
- If the reference runs there while the substitute rejects the language feature,
isolate a general capability reproducer and open or update
FIX_STATUS.md. Use language semantics, not reference design values, to specify the tool fix. - If the reference also fails under the supporting tool, retain both logs and re-triage tool/environment and dataset causes; failure alone proves no floor. Do not feed an expected answer back to the author.
Capability example: a testbench's break; is rejected by stock iverilog 12. The golden compiles and passes under Verilator with
--timing and under the forked iverilog, so it was a confirmed tool gap and is
now fork-closed.
Never patch a tool to "pass benchmark X". That is the over-fit prohibition:
the fork dissolves a CAPABILITY floor, never the honesty boundary. A genuine
algorithm-hard port that is honestly DEFERRED in FIX_STATUS.md is a
known-deferred engineering item, and still not an unfixable floor.
A plain tool-substitution gap is NO LONGER T5 by default. Since we fork the
tools it is Category D FORK-FIXABLE; it may be recorded as a real floor ONLY
once it carries the deferred FIX_STATUS.md entry AND has passed the
FLOOR-proof. Never "AI cannot".
FLOOR-proof: assess a floor claim with native controls (#724)
Across one campaign FIVE residuals were labelled "benchmark defect" and ALL FIVE were later overturned and PASSED — each was our own off-by-one, interface or reset-boundary bug. A floor claim is almost always a misread, and the asymmetry is severe: a false floor ships an unfixed real bug, while the disproving run is cheap.
A residual may be labelled FLOOR only after this three-step FLOOR-proof, in order:
- Run the exact official scorer on the candidate RTL — the real harness, toolchain and required image. Preserve native failure logs in the scoring environment. Authors/reviewers continue to read only supplied INPUT; a failing hidden assertion cannot define a repair requirement.
- Use the ORIGINAL, unmodified reference through the SAME scorer as an authorized host-side control. Preserve its actual verdict separately from the candidate. Original failure can expose a scorer/environment defect; original success shows a difference, not that the candidate violates INPUT. Neither outcome tells an author what behavior to implement.
- Only then assess FLOOR, with the prompt clauses, exact official failure evidence and controlled environment. A confirmed contradiction needs two mutually-exclusive requirements for identical stimulus derived from INPUT; a scorer's contrary expected value is not a source requirement. If the cause is still unproved, retain an unresolved diagnosis, not a floor.
Variant — the harness contradicts the given context itself. When the contradiction is provable from the given input alone, the floor-proof is (a) a cross-round expected-vs-got table whose expected values come from the supplied mapping, shown stable across at least two independent authoring rounds so it is not a one-off; and (b) a replay under the design's OWN input-derived testbench checking that mapping. Keep the official disagreement in the host-side score. If INPUT is silent or ambiguous, AI authoring names that ambiguity instead of learning a value from the hidden oracle. Consistent candidates alone do not prove the dataset defective; preserve the unresolved alternative causes.
why_not_bucket_a: deciding whether two assertions are truly mutually exclusive for identical stimulus is an open-ended meaning judgment no regex makes. The deterministic half — a floor claim MUST carry the host-side original-RTL control and exact official failure evidence — supports that diagnosis, never a requirement inferred from the reference answer.
Capture enhancement
Post-score recovery is not part of the blind score. When AI finds a correct repair that Program missed:
- isolate the general discriminator from prompt and RTL evidence;
- add a benchmark-agnostic program rule/check/fix at the normal product enforcement point;
- prove the new test fails on the unfixed product and passes after the fix;
- run no-leak and false-positive controls on non-benchmark/general inputs;
- submit the general fixes through the owner's landing workflow and wait for every required fix to land; branch-only hand-in does not authorize a PR;
- refresh main and the installed plugin/MCP, then run ONE new complete, memory-off clean-room cohort. Do not rerun intermediate trees before landing.
Do not encode problem IDs, dataset labels, hidden expected values, or a lookup table. Capture is successful only when the next ordinary Program First run produces the right behavior without benchmark-specific knowledge.
RESULT requirements
Every result must include:
- Summary/status and exact official score;
- whether measurement was before v1.41.82, at/after v1.41.82, or
VERSION_NOT_RECORDED, with the measured plugin version/commit. Disclose earlier results as pre-cleanup; do not relabel them input-only or rerun merely to change the label. Keep the boundary explicit in historical comparisons; - the best earlier official raw score for the same problem population,
denominator and counting method, beside the current result in the first
sentence. Start that sentence with
REGRESSIONwhen the current score is lower. Keep local fixed-tree controls distinct from product scores; - Program-solved and AI-solved counts beside their recorded earlier counts.
Mark missing historical attribution
NOT_RECORDED; do not infer it from a model name. Program-solved means official PASS of a Program-authored candidate. A legitimate source-ambiguity handoff with actual AI_BACKUP authorship receives AI credit, not Program credit or AI_REPAIR attribution, and is not a Program regression. Report that transfer separately from actual Program regressions; an AI recovery of a proven Program defect does not erase that regression. Apply the owner's official-score and Program-count non-regression targets before declaring the requested evaluation complete; - clean-room/blindness statement;
- environment and official scorer evidence;
- full per-problem four-stage CSV/JSON path and reconciled aggregates;
- all failures with evidence-backed triage;
- captured enhancements/PRs and what they do not claim;
- reproducible next action.
Historical comparison is host-side result reporting. Read saved verdicts, counts and input identities, never old answers into the fresh authors' context. Do not change the denominator, exclude failures, or splice previous candidates to reach a historical target.
No RESULT means the run FAILED. Never publish a number without the result that backs it. The load-bearing failure is the launch-and-idle abandon bug: the scorer or runner finished, a verdict and artefacts exist, and no RESULT was ever written because the turn ended while the run was detached. To keep your turn alive to completion, drive the long run through the BLOCKING supervised waiter that returns only on exit or stall — never a detached fire-and-forget.
Why the instruction alone loses to a plausible model (#558). An agent ended
its turn on a still-running flow, saying "I'll yield until the harness
re-invokes me when it exits". Nothing did. claude -p is one-shot, so the
three beliefs that justified it are each impossible:
- "the harness re-invokes me when the background job exits" — nothing re-invokes a finished turn. There is no such mechanism.
- "a background waiter is armed to fire" — a waiter can only wake a turn that is still alive. It cannot start a new one.
- "the monitor will fire" — a monitor notifies the DISPATCHER, not you. It cannot resume you.
Yielding does not pause your turn; it ENDS it, and the "then write the result" step never runs. The rule above is not a preference about style — it is the only way the deliverable gets written.
Handoff
End with ## Summary or ## Verdict, followed by a clear Next: statement.
Common product input
Stage visible design documents and supplied RTL through the thin IO adapter. The unchanged Router decides the task's native entry and evidence exit. A supplied-RTL continuation resumes at validation within that same evidence window. The public entry always executes the native step table.
python3 programs/benchmark_dispatch.py BENCH --solve --dataset VISIBLE_INPUTS --run FRESH_RUN
python3 programs/benchmark_dispatch.py BENCH --resume --dataset VISIBLE_INPUTS --run FRESH_RUN
Confirm input-only AI routing before Program authoring. Preserve independent blind review, prompt-derived challenges, AI backup, and official scoring. Native measurements and reaching a boundary alone are not a benchmark score.
Captured rule: keep the actual transformed candidate
When a layout transformation completes with valid State views and a native
count over its original limit, retain its produced candidate and report FAIL.
Later tools measure that candidate. A transformation that enters Step.run
but fails without a valid output State clears its declared outputs, including
views declared as both inputs and outputs. A refusal before Step.run
preserves shared inputs while clearing output-only v
…(truncated)