testbench-gen — deterministic professional testbench generation
The canonical flow's Step 4 (Simulation) derives a professional, high-coverage
testbench from the Phase-1 layers — never a hand-waved skeleton. The load-bearing
work is a PROGRAM: programs/professional_tb_gen.py (exposed as MCP tool
eda_professional_tb). This skill is the judgment layer around it.
What the program does (program-first)
professional_tb_gen.py <project> reads the project's phase1/generated_docs/L*.json
and emits, under phase2/stage1/sim_professional/<top>/:
tb_<top>.py— a cocotb testbench: clock/reset from L8/L9, a reference model, a SCOREBOARD, functional coverage (cocotb-coverage covergroups), and a directed-corner + constrained-random stimulus loop.<top>_coverage_model.json(an L28 coverage model),<top>_assertions.sva(an L29 assertion set), a cocotbMakefile, andverification_plan.json.
It combines open-source frameworks rather than re-inventing them: cocotb
(TB base) + cocotb-coverage (functional coverage + constrained-random) +
Verilator/Icarus (sim) + an SVA bind for Verilator/SymbiYosys. Add
cocotb-coverage to the runtime (see mcp-eda/INSTALL_GUIDE.md).
The reference-model strategy — 3 tiers (honest, never a vacuous pass)
- Closed-form (arithmetic primitives
c = a OP b): reusesarith_oracle_tb_gen.compute_golden— the single source of truth. - Bounded-latency streaming scoreboard (serial datapaths — one parallel
operand + a 1-bit serial operand + a 1-bit serial result, e.g. a bit-serial
multiplier): the scoreboard AUTO-DERIVES the output
(latency, bit_order)from a calibration vector, then checks every vector against the closed-form reference. This closes the serial-datapath DEFER thatarith_oracle_tb_genpunts on entirely (proven: spm bit-serial multiplier, 208/208 vs(x*y) mod 2^N,order=lsb latency=1auto-derived). - Reference hook (classes with no closed form — CPU/SoC/DSP): the generated
TB emits a clearly-marked
reference_modelhook thatTestSkips until filled — NEVER a silent vacuous pass. The judgment layer (this skill / spec-to-refmodel) fills it from L10 vectors or a spec-derived model, with the program-vs-AI cross-check (dual-track convergence).
When YOU (the judgment layer) act
Fill a Tier-3 reference model: read the L-docs (L2 function, L3 protocol, L10 vectors) and author the Python reference the hook needs — then re-run so the scoreboard is real. This is the only place an LLM is needed; the interface, clock/reset, coverage bins, and (for Tier-1/2) the reference are all derived by the program.
WRITE IT TO
expert_reference_tb.py, NOT TOtb_<top>.py.phase2/stage1/sim_professional/<top>/expert_reference_tb.pytb_<top>.pyis REGENERATED byprofessional_tb_genon every Step-4 run, so an oracle written there is deleted by the very re-run the gate asks for —cpu_functional_oracle_waiver_checksays "fill … with the testbench-gen expert fallback and re-run Step 4", and that loop has no fixed point if you edit the generated file. The siblingexpert_reference_tb.pyis deliberately stored apart so regeneration cannot erase your work;professional_tb_gen._load_expert_reference_tbreads it and promotes the run todut_kind=expert_reference, which is what makes the Phase-2 runner actually launch cocotb for an otherwise-generic class.For the loader to ACCEPT it (it rejects placeholder shapes before a simulator is launched), the file must:
requirement why a @cocotb.test()coroutine that OBSERVESdutand carries a realasserta testbench that never reads the DUT is not an oracle print PROFESSIONAL_TB PASSfrom a passing checkthe marker the Phase-2 runner greps for contain no TestSkipa skipped hook is the unfilled state The JUnit is still the authority: the run is credited only on
tests>0, passed>0, failures==0, errors==0. The marker never credits a case on its own.In a benchmark run, an AI backup task whose evidence window reaches Step 4 names this path in
reference_model_handoff. Fill it before--resumeso the first re-gate measures it. A model filled or replaced later is measured by the next--resume, which re-enters the runner on the same RTL bytes and issues a fresh review. A model the last run already measured is not re-run.Measured (2026-09-02, a subservient × gf180mcuD acceptance run): the whole convention was named in exactly two files —
professional_tb_gen.pyand its own test. No gate message, no skill and no doc mentioned it, so an agent following the gate's own instruction wrote a full 7-oracle testbench intotb_<top>.pyand watched the next Step-4 run delete it.Review coverage: after a run, read
coverage_<top>.xml+ the L28 model; if functional bins are unhit, add directed vectors or tighten the constrained-random constraints. Loop until the closure policy (functional bins 100% + code coverage targets) is met.Formalise assertions: the L29 SVA carries reset→known-state + protocol handshake properties deterministically; L16 must/shall prose is emitted as
TODO(spec-to-assertion)stubs for you to formalise (never fabricated as a passing SVA).
Skill: Reuse the reference model with physical propagation delays
When to apply: an expert reference testbench is replayed against an implemented netlist or SDF annotation.
What to do: preserve the clock frequency, transaction sequence and expected
cycle behavior. For RTL and gate-level benches, drive inputs at the SDC
set_input_delay instants. Sample each output at its capturing clock edge minus
that port's SDC set_output_delay -max, using each bit's own delay for indexed
constraints. A zero input delay drives immediately after the active edge in the
same time step through a non-blocking assignment. If no output delay is declared, sample
just before the next active edge. Never use a fixed offset after the launching
edge or choose a sample time from the DUT's measured propagation delay.
Prove asynchronous reset while the clock is held stopped: establish idle inputs, assert reset, allow propagation, check the specified reset outputs, and check that they remain reset without any active clock edge. Then release reset and restart the clock according to the input contract. Do not replace unknown outputs with zero or suppress native SDF diagnostics.
For configuration that commits at a period boundary, wait for the previously active period as well as the new period before checking a settled configuration. Keep the separate in-flight update test that checks the current pulse is not truncated. RTL and gate-level replay must both pass; a reference-model correction does not change an earlier native result.
Skill: Software-visible field profiles for reference models
Pattern: An instruction extension or interface selection is treated as proof of every software-readable or writable field, so a reference model confuses internal saved state with the selected software interface.
When to apply: Authoring a Tier-3 model around an implementation that exposes a documented subset of architectural or interface state.
What to do: First record the declared scope, selected IP revision and pinned public capability sources, then distinguish software read masks, write permissions and side effects from internal state. Compare only source-supported fields while preserving transaction widths, timing, full byte commits and unknown-value checks. Check an internal saved state through its specified observable effect when software readback is not promised. An unsupported or ambiguous requirement remains NOT_MEASURED and cannot be silently removed from required acceptance. If a newly reviewed profile narrows an earlier oracle, preserve the old oracle and counterexample and label the new scope explicitly; never choose masks or goldens from DUT traces. Retain positive, fault-specific negative and byte-identical reverse controls.
Worked pattern (anonymized): An interrupt implementation may retain a saved enable internally for return behavior while exposing only the current enable through software reads. Test the guaranteed read field and the specified return effect independently; do not invent read or write access to the saved field. A second eventual interrupt alone does not prove that pending state survived masking: use a discriminating countercontrol or keep that property unmeasured.
Why this is GENERAL: The same distinction applies to processor status fields, peripheral register views and protocol capability subsets. Instruction or interface membership, internal state and software access are separate claims; actual native/JUnit evidence remains scoped to the tested fields and effects.
Captured by benchmark-enhancement-capture 2026-10-15.
Skill: Mode-conditioned meaningfulness in reference oracles
Pattern: A software-visible field is explicitly meaningful only in selected modes or states, but a Tier-3 reference oracle asserts one exact value unconditionally and rejects another source-permitted implementation.
When to apply: Authoring or reviewing a full-stack expert reference when verification_plan.json carries mode_conditioned_field_profiles, or when INPUT describes a register or protocol field as not meaningful, undefined, or don't-care under a named condition.
What to do: Build a field-by-mode profile from the cited INPUT statement. When its condition holds, do not assert an exact value for that field. Continue to check every source-defined peer field, transaction timing, known-value requirement and observable effect. In each mode where the field is meaningful, retain its complete source-supported behavior. Keep two legal-value positive arms for the non-meaningful mode, a meaningful-mode negative control, and a reverse mutation of a neighboring defined field. Derive neither the condition nor an expected value from DUT traces.
Worked pattern (anonymized): A status bit reports lost output in manual overwrite mode but is explicitly not meaningful in automatic stall mode. Automatic readback zero and one are both legal for that bit. Manual loss behavior, output-valid, idle and stall remain checked. Removing all checks for the register is a vacuous repair and must fail the peer-field reverse control.
Why this is GENERAL: Mode-dependent status and capability fields occur in accelerators, peripherals, CPUs and mixed-mode controllers. The producer emits source-cited profiles without a design name, address, bit number or inferred trace value; the expert review applies them to arbitrary Tier-3 oracle code.
Captured by benchmark-enhancement-capture 2026-10-21.
Skill: State-local control independence
When to apply: a protocol or FSM uses different controls in different phases. Derive a dependency table from the admitted input: which controls can change the next state, stored data and outputs in each phase?
What to do: exercise both values of relevant controls and verify that
irrelevant controls do not disturb progress. If unknown/high-impedance controls
are legal in that phase, also drive X/Z on only the irrelevant controls while
keeping relevant controls known. Include an unchanged binary control and a
relevant-unknown negative/default control. Do not invent an initial state,
reset, undefined-input output value or universal four-state requirement.
For a complete explicit Moore table, state-local Boolean support reduction is
enforced by programs/moore_fsm_table_emit.py; an incomplete table must still
refuse emission. Apply the behavioral dependency checks to AI-authored RTL too.
Why not Bucket A: the complete-table optimization is already a program. Determining legal unknown-input domains and dependencies in arbitrary prose requires source-bound judgment. A binary test or a port-reference coverage number alone does not establish phase independence.
Optional reset-retention byte stimulus
When a retention test needs a designer-chosen memory stimulus, declare
verification_stimuli.reset_retention.memory_preload in the project's
plugin_output/declaration.json through spec_declaration_emit --from-json
or --set. This verification choice is separate from the SRAM timing contract.
It is a non-empty list of {"address": 0, "byte": 165} entries: exactly those
two keys, unique addresses within the selected memory depth, and integer byte
values 0..255. Boolean, string and floating-point values are refused.
testbench_gen.emit_unit_tbs applies the bytes after initializing every memory
location and before observing either reset arm. It preserves every rising-edge
write, the pending first-edge check, all-byte comparisons and the 64-cycle
activity budget. Absent preload keeps the stock seed; malformed preload cannot
credit actual-byte retention. A chosen stimulus alone is never execution
evidence: absent non-idempotent pending activity remains NOT_MEASURED.
Declared serial arithmetic scenarios
The existing unit producer can generate operand-boundary, consecutive-transaction
and synchronous-reset scenarios from the case's own L10 scope. It requires the
selected serial function, bit order and settled observation offset, plus
transaction_rearm, rearm_reset_cycles and
parallel_hold=through_final_product_bit. Supported rearm is clocked_reset,
or continuous when the settled offset is zero. Unmodeled input roles, clock
domains, reset styles and ambiguous scopes remain unmeasured.
The scenario oracle uses independent modulo goldens at one declared window; it does not search DUT alignment to choose an expected answer. It exercises all selected corner pairs, the stated transaction population, early/middle/late interruptions with single/multiple reset holds, and between-edge reset pulses. A row listing multiple supported families executes each family in the same native case, with separate actual populations and an aggregate comparison total. Generated scenario credit requires the current native comparison population and unchanged input, DUT, declaration, TB and transcript identities. Generic generated arithmetic remains insufficient for scenario credit. Authored and delivered testbenches retain their original priority.
latency_cycles_origin=settled_edge_offset is the normal CR-5 convention.
Explicit legacy pre_nba_sample_offset is normalized once. Native manifests
carry their origin; a bare measured integer is not origin evidence. Neither
convention relaxes an incorrect declared window.
Capture
Any recovery here (a new reference-model pattern, a new coverage-derivation rule,
a streaming-alignment case) is captured per benchmark-enhancement-capture.
Route full-stack professional-testbench recovery to professional_tb_gen.py.
Route an exact-name L10 .v / .sv unit oracle from
testbench_gen.run_unit_tbs through the canonical
phase2.l10_unit_tb.testbench_gen.run_unit_tbs entry: Bucket A belongs to
testbench_gen.py with _l10_execution.py as its receipt owner, and Bucket B
belongs in this skill. Do not borrow the full-stack route for a unit-TB record.
For L10 unit-oracle recovery, keep the authored TB bytes and bind the complete source closure needed to reproduce its expected result. A wrong expected value or wrong stimulus must still compile and then fail the native comparison; the restored original bytes must pass through the normal producer and write a fresh execution receipt. A broad reference-suite PASS cannot credit a named vector it did not execute. If the gap is in a forked simulator (Verilator/Icarus/cocotb), also emit a Bucket-T forked-tool item.
Gaps to grow into (from the 2026-07-11 verification research)
- Fill the currently-empty L22 verification plan fields (coverage_goals / formal_properties / regression_matrix) and wire them to L28/L29.
- Add L30 stimulus/constraints (constrained-random weights, sequences).
- Widen code coverage to FSM-transition / expression / condition (Verilator
--coverage-user) and gate sign-off on it. - pyuvm structural env + riscv-dv for CPU-class constrained-random.
Skill: Legal delayed acknowledgement withdrawal with public observation
Pattern: Nominal service can complete while leaving a source-permitted request-low acknowledgement tail untested.
When to apply: Native coverage identifies that branch and the original interface and reset prose permit bounded delayed withdrawal with stable payload.
What to do: Read the complete handshake and reset obligations before choosing a finite transaction schedule; do not transfer a reset-only timing requirement to every transaction. Admit the normal design class and expert-reference contract first. Give one owner to the public service coroutine, use a separate public-pin monitor to check request remains low and acknowledgement, validity and payload remain stable throughout the selected tail, then withdraw acknowledgement and restore the existing service even on failure. Check subsequent progress against an independently specified result from the input, never an expected value inferred from the DUT. Keep the original vector and control populations, their order and goldens intact; append a named case in the persistent expert source with its complete source closure. A wrong independent expected result must compile and reach a real native comparison failure; exact restoration must pass. A zero-tail control may isolate coverage causality but does not qualify delayed-tail behavior. Hand the complete source back to the normal professional generator and native/JUnit consumers for fresh full-population measurement.
Why not Bucket A: The interface prose permits several finite service schedules and does not uniquely select an observable progress witness. Those choices require source-bound engineering judgment; existing programs retain source, population, execution and coverage validation.
Worked pattern (anonymized): A delayed-tail case passed with a separate public monitor and an input-derived functional golden. Changing one expected result caused a native comparison failure and restoring the exact source passed. An identified native branch counted three, zero and three in delayed-tail, zero-tail and restored arms without changing the coverage denominator. A driver delivery count alone did not prove internal one-time consumption.
Why this is GENERAL: Applies to request/acknowledgement interfaces with source-supported payload stability, legal bounded timing and independently observable progress. It establishes the measured public behavior and selected branch, not complete internal consumption, clock-domain signoff or whole-design qualification.
Captured by benchmark-enhancement-capture 2026-10-21.
Bucket B — skill sections to APPEND to skills/testbench-gen/SKILL.md
Skill: Derive event phases before an executable timing challenge
Pattern: A timestamped table is silently treated as either pre-edge input values or post-edge drives, so an executable challenge shifts acceptance and completion by one cycle while appearing precise.
When to apply: Reviewing a generated reference model, waveform replay or proposed executable challenge whose visible prose and timing rows must jointly define synchronous behavior.
What to do: Keep Program First: run professional_tb_gen.py through the normal producer and freeze its exact output and raw evidence. An independent IC expert then derives from the whole visible contract the input drive phase, capture edge, settled sample point, output-valid interval, transaction overlap, reset behavior and any condition-specific don't-care. Cite the source for each decision and distinguish requirements from assumptions. Where a timestamp admits pre-edge values and post-edge drives, explain both schedules and their observable consequences; choose the best source-bound interpretation using all visible prose, diagrams and tables before declaring ambiguity. Neither a table timestamp alone nor simulator scheduling, a DUT trace, prior score or hidden oracle supplies normative authority. Encode the chosen phase explicitly in the challenge, avoiding a drive/sample race and observing settled outputs. A disagreement still needs a prompt-derived executable proof that fails the frozen Program output; repair must pass the same unchanged challenge, normal gates and a fresh independent review. Do not apply a universal one-cycle correction or grant automatic PASS. If independent adjudication leaves two interpretations equally supported, preserve both and report the affected timing claim NOT_MEASURED with the exact missing convention; keep separately determined checks in scope. This review guidance prevents an error and does not itself prove a functional recovery.
Worked pattern (anonymized): A neutral contract has a 10 ns clock, a request row at a rising timestamp of 40 ns, and completion two full periods after request acceptance. If the row describes the value present before the edge, acceptance is at 40 ns and completion at 60 ns. If it describes a drive after the edge, acceptance is at 50 ns and completion at 70 ns. Visible prose saying the request is stable before the listed edge selects the first schedule; a table without that statement requires the rest of the contract to adjudicate. Define the settled observation point and valid interval in either schedule instead of shifting every expectation by one cycle.
Why this is GENERAL: Timestamped synchronous specifications occur in controllers, pipelines and streaming interfaces. The reasoning concerns event semantics and source authority, not any design identifier or stored solution. Existing generators and execution consumers retain their proof responsibilities.
Captured by benchmark-enhancement-capture 2026-11-02.
Skill: Close instruction coverage through independent public effects
When to apply: native branch coverage is below its original floor even though every authored processor test ran and passed. Read the raw uncovered points and the declared instruction profile before changing stimulus.
What to do: extend the persistent expert_reference_tb.py with the missing
source-supported encoding families. Keep disabled extensions disabled; test
their required rejection instead of enabling them to reach decoder branches.
Construct instruction encodings and expected public transactions before DUT
execution. Exercise compressed register, stack, memory and control forms when
the profile admits them. For an illegal instruction, check the source-defined
trap cause, precise instruction address, faulting encoding and resumed progress.
For a call, observe both the link value and a destination marker: a correct
return address alone does not distinguish a jump from fall-through execution.
Preserve every existing program, expected result, test and coverage point.
The native JUnit results and unchanged coverage limits decide closure.
Measured scope: an RV32IMC hard-macro public prefix on stock image 0.4.81 ran all 23 original cases successfully but hit only 374 of 632 branch points. Five added instruction-family cases retain those original programs and use 934 independent public bus comparisons per service mode. The matched native after passes 28 of 28 tests and hits 447 of the same 632 branch points, 70.73% against the original 70% floor. All eight public-prefix rows pass with no newly failed rows. This is a verification-stimulus repair; it does not establish whole-route signoff or a PPA improvement. The native coverage reader and professional testbench handoff required no program change.
Skill: Test eligible interrupt priority and distinct vectors
When to apply: a reference model checks interrupt arbitration or vectored dispatch.
What to do: derive the winning source from the input's stated priority policy. Assert competing requests together and enable every contender in both its individual mask and the global mask before claiming priority coverage. A priority test requires at least two simultaneously eligible sources. Exercise several source positions so one correctly handled pair cannot conceal a reversed encoder.
Give unselected vector entries a distinct, observable wrong-handler result. An empty vector table can fall through no-op instructions into the intended handler and make its prologue appear correct. Check dispatch and reported cause separately. Compute expected winners and handler effects before running the DUT; never choose them from a trace. Preserve a source-derived failing case and its native corrected result, and keep any remaining coverage shortfall explicit.
Why this is general: interrupt controllers and event routers differ in priority and masking policy. The input supplies that policy; the test must establish eligibility and distinguish the selected destination.
Captured by benchmark-enhancement-capture 2026-11-07.
Skill: Complete the declared expert reference fallback
Pattern and application: A skipped professional testbench can expose an unfilled expert reference hook without proving an RTL defect. When professional_tb_gen.py reports NOT_MEASURED with a testbench-gen fallback, inspect the actual skip cause and producer evidence. The IC expert resolves evaluation semantics from the visible contract instead of asking the human to supply the model.
Model derivation: After the producer's typed handoff, fill only its declared expert_reference_tb.py with an independent input-derived model. Derive operand and result widths, signedness, overflow and flag ordering, reset behavior, deassertion and settled observation phases from that contract. For a wrapping result, determine whether the flag observes the width-limited value; for selection, establish the specified bit significance. Check source-promised known values and latch behavior after deassertion: an X-only check can miss a wrong known value. Expected results must not come from DUT behavior or a hidden oracle. Preserve the prior candidate, receipts and reviews.
Worked pattern and scope: An absent hook produced one genuine JUnit skip and no passes or failures. Filling the declared hook with an independent model allowed the same frozen RTL to execute; substantive wrong-behavior controls failed without skipping and exact restoration passed. These are component proofs. Rerun the normal producer and require actual non-skipped JUnit comparisons, its success markers, discriminating wrong-behavior controls and a fresh independent review. Retain required gates and do not perturb RTL to force entry. Model derivation from arbitrary visible prose requires engineering judgment; existing programs own handoff and execution validation. Component evidence cannot credit canonical handoff, regating, whole-flow acceptance or a benchmark score. Captured by benchmark-enhancement-capture 2026-11-02.
Captured rule — Input-required paired verification needs both operations
When the input affirmatively requires paired or round-trip verification, a
passing forward-only simulation does not establish that obligation. Compare
the original requirement with L10 and the actual generated sequence before
accepting the functional result. The existing typed request is round_trip: true
with the cited input sentence in round_trip_source; if it is missing, return
the requirement to the Phase-1 author instead of inferring intent from a name,
a keyword, or an optional or prohibited operation. Confirm that the native
sequence actually exercises the reverse operation and checks the recovered
source data. Missing input-declared operation encodings remain unresolved;
neither a forward-only PASS nor code coverage supplies the missing evidence.