vinta Prompt experiments
Use the existing helpers in the command reference. Keep helpers and durable round records under scripts/prompt-experiments/, and raw output under tmp/prompt-experiments/. Applying results to production is a separate task; leave production source and prompt tests unchanged.
Resume or start
A round contains many candidate iterations. Start with its current-state entry in REPORT.md and the relevant frozen prompt. Search older reports for the current failure or hypothesis, then read only the matching findings. Load corpus text, failed answer rows, or reference sections when the next decision needs them. Reuse recorded checks and diagnostics while their inputs remain unchanged.
For a new round, read the destinations and round record sections once, inspect the production prompt and question builder, and select verified corpus records with their exposure history. Record the goal, selection, controls, and reused findings. Declare holdout coverage and separation rules now; collect holdouts only after confirmation passes. Existing unseen reservations may remain private.
For an existing shape, measure fresh shipping as the control; for a new shape, start with a concise instruction and explicit labels. Screening uses corpus order and a recorded seeded shuffle, with 1 attempt per target per order. If no development failures remain, collect more verified coverage or finish without a prompt change.
For an audit of a completed round, read verification and retention and inspect its saved evidence; a new inference run is new evidence.
For inference on a configured machine:
- Use
scripts/prompt-experiments/.env.local. Reuse the livepangu-evalconnection, or attach it to the configuredPANGU_CDP_URLwithplaywright-cli -s=pangu-eval attach --cdp="$PANGU_CDP_URL"after loading the settings into the environment. Try the saved connection before inspecting Chrome settings or rediscovering local values; a successful attachment completes connection setup. - Apply the reuse conditions to the recorded checks. Run the missing or invalidated checks and record any missing runtime metadata.
- Once those checks pass, run shipping and the first frozen candidate with
sweep.mjs; resume an existing round from its recorded next comparison.
Read setup and repair for a missing prerequisite or failed connection/runtime check. Follow the relevant section, then resume the interrupted step.
Checks to reuse
| Work | Run when |
|---|---|
| Runner tests | No matching passing check is recorded, or runner, gate, or relevant dependencies change; add coverage for helper changes |
| Production fixture replay | No matching pass is recorded, or fixtures, annotations, relevant spacing/detector/extractor code, replay tooling, or browser version/configuration change |
| Manual runtime evidence | Missing evidence, reconnect, or changed browser/profile/model, loaded checkout, or production sampling |
| Fresh screening baseline | New round; changed scored records, example-page exclusions, gold, production inputs/edits, baseline prompt/options, tooling, or schedule; browser/model restart, update, reconnect, or uncertain runtime continuity |
Use the first sweep.mjs invocation as the preflight for cases, rendered prompts, extension worker/profile, and model readiness. Its output records those checks, browser/Node versions, and requested sampling/schema options. Read that evidence; inspect a failed check only when repairing it. Use --check for offline work or diagnosis. Helper-managed output enforces ignored, untracked destinations; apply destination checks only to private writes that bypass it.
Reuse a complete baseline's saved answers only within the same round and continuous browser connection with unchanged model/runtime and controls above. The runner still creates fresh inference sessions. Record which baseline each screening comparison uses. Gates check orders and sampling, not browser/model identity; if continuity cannot be established, run shipping with the candidate. Match production sampling, language declarations, and schema handling.
Corpus and roles
Use verified real text for examples, diagnostics, development, confirmation, and holdout. Preserve authored text and context; synthetic fixtures are only for deterministic implementation tests. Read source collection for new/changed records, evidence gaps or disputes, and live-page behavior. Reuse verified snapshots; regenerate affected production inputs and spacing from saved context when the contract changes.
Before scoring, every eligible target needs a verified label and individual/combined spacing. Keep routing and authored-space exclusions separate from classifier accuracy. Disputed meanings stay unscored; gold unsure needs an explanation of why the exact production context is insufficient.
Keep canonical pages disjoint across examples, development, and holdout. Record exposure; uncertain exposure belongs in development. Promoting a development case to an example excludes its whole page from scoring and requires a fresh baseline. Keep holdout text and labels outside the prompt-editing context until frozen evaluation. If holdout evidence guides tuning, end the round, move affected pages to development, and reserve fresh holdouts.
Screening loop
- Record the hypothesis, predicted effect, parent variant, and one changed prompt dimension. Freeze the variant ID, prompt bytes, and question builder in
prompts.mjsbefore inference. - Run the candidate on the same scored records, orders, and attempts as its baseline. Use saved baseline answers when eligible; otherwise run both prompts. For hyphen-digit, run the paired gate command on saved answers to check labels, individual/combined spacing, and the frozen protocol. Other shapes require a compatible shared evaluator with production edits. A sweep exit or
--require-perfectis not a paired-gate result. - Read CLI scores, miss IDs, unstable cases, and gate summaries first; inspect only relevant failed rows. For a new unexplained development failure, use
--diagnosticswith 1 order and 1 attempt: follow classification with translation, target identification, and label-meaning questions. Reuse earlier diagnostic findings for the same failure. Explanations may be rationalizations and never count as accuracy evidence. - Append the result and decision to
REPORT.md; update its current-state entry with the next hypothesis and any invalidated checks. Prefer the smaller prompt when qualifying candidates behave equally.
After two or three similar wording changes leave the same failures, investigate another cause. Revisit Gemini Nano’s diagnostic interviews from step 3 and ask targeted follow-up questions where clues remain unresolved. Use those clues to search current primary sources online for materially different approaches tied to the observed failures. Record each source alongside the hypothesis and predicted effect, then test one changed dimension through the existing gates. Use sourced examples for an identified vocabulary/domain gap, revisit disputed annotations, or report routing/extraction limits. If no credible new hypothesis remains, end the round with the bottleneck and supporting evidence recorded. Finish the iteration with a recorded gate outcome and next decision; retain raw artifacts without loading the full runs into context.
Confirm and finish
Once one candidate meets screening and the task's requirements, read confirmation and holdout. These phases always require fresh paired comparisons. Prepare new holdouts in a separate context only after confirmation passes, keeping the declared coverage and page separation fixed.
When ending the round, read verification and retention, complete the report, close temporary inspection pages, and disconnect. Failed or inconclusive experiments are valid outcomes; later dependent phases remain unrun.
Acceptance gates
A correct attempt has no error, returns the expected label, and produces expected individual-target and combined-excerpt spacing through production edits. Missing attempts and skips fail. A case passes only when every scheduled attempt passes. Report label and spacing correctness separately.
| Phase | Required result |
|---|---|
| Preflight | Every model input/example has verified real provenance and production context; role separation passes; no unresolved labels in scored sets |
| Screening | Candidate preserves every baseline-passing case and fixes at least one baseline-failing case outside its example source pages; no inference errors in either run |
| Confirmation | The same rule holds independently in both fresh runs, and at least one same scored case is a stable improvement in both; no inference errors in either run |
| Holdout | Candidate preserves every baseline-passing case and has at least as many passing cases as baseline; no inference errors in either run; a new improvement is not required |
For baseline-unstable cases, require candidate correct-attempt counts to be no lower in each matched run. Aggregate gains cannot offset a new case failure. List remaining failures and coverage gaps. Report target, sentence, page, and publisher counts separately; repeated answers are not independent source cases.
An infrastructure failure leaves the comparison incomplete; retain its artifact and record it. After repair, repeat the entire affected comparison under the frozen protocol, refreshing the baseline if required above. Otherwise report it incomplete. Never retry a semantic failure until it passes.
References
Chrome's evaluation guidance describes evaluation layers, contamination from prompt examples, and final evaluation on unseen cases. Its suggestions to generate synthetic data do not apply to this workflow.
For prompt-only changes, use the checked-in runner and its recorded API options. Read current API documentation when changing API-dependent code/options or diagnosing an unresolved API failure: Prompt API for model sessions and structured output for schema handling. Research prompting ideas when a stalled hypothesis needs new evidence.