Nobrainer Autoimprove

NoBrainer Autoimprove

Improve one artifact only when a frozen evaluation proves a candidate better. This independent adaptation applies Andrej Karpathy's autoresearch measure-change-keep loop to skills, prompts, instructions and other reviewable artifacts; it is not an official Karpathy project.

Route the smallest fix

Fix an obvious typo, dead link or deterministic defect directly and cover it with the normal test. Do not start an experiment when the objective cannot be measured, target/evaluator isolation is impossible, or the change needs an unapproved production, credential, publishing, safety or irreversible action.

Treat owner correction as evidence, not permission for a global rewrite:

• task-local clarification stays in the task;
• a repeatable behavior gap becomes a regression scenario;
• a verified durable fact or preference follows LEARNING_WRITE_POLICY: AUTO_SCOPED writes one minimal sourced rule to one canonical store, ASK prepares one exact diff, and OFF keeps it task-local: do not create a durable diff in AGENTS.md, tasks/lessons.md or a wiki.

On AGENT_ERROR_CORRECTED, preserve the failing scenario and error fingerprint, fix the active result first, then test the smallest reusable prevention. One anecdote never justifies a broad user trait or unrelated write.

Frame before mutation

Freeze this contract before changing the target:

TARGET: <one artifact and allowed write scope>
OBJECTIVE: <one observable improvement>
BASELINE_VERSION: <commit/hash/copy>
DEVELOPMENT_EVAL: <representative iteration cases>
HOLDOUT_EVAL: <unseen promotion cases>
HARD_GATES: <format, safety and deterministic checks>
SCORE: <criteria, aggregation, direction and meaningful delta>
EVAL_INTEGRITY: <separate owners, access and calibration controls>
SCORE_RECEIPT: <append-only trial identity, provenance, raw evidence and decision>
NOISE_POLICY: <paired repetitions, dispersion and retry rule>
BUDGET: <finite variants, rounds, repetitions, retries, time and resources>
PLATEAU: <minimum delta and dry rounds>
ROLLBACK: <retained baseline, restore and readback>
OWNER_GATES: <action, owner and recorded evidence>

Preserve the baseline in an isolated copy, branch or worktree. One coordinator owns the champion and log; candidates never race on the same file.

Return a complete initial plan

An initial plan comes from a completed experiment contract, not a summary of this skill. Resolve every field internally; the owner-facing plan need not mirror internal labels. Never omit a hard gate or replace exact evidence with "immutable receipt", "bounded budget" or similar shorthand.

The plan must operationalize:

• an observable access boundary: candidate write scope excludes evaluator inputs, labels, logic, extractor, cases and holdout;
• ordinary positive, known-bad and tailored deceptive controls for the literal gaming, omission and unsafe-abstention risks;
• fixed baseline/candidate identity, exact paired run count, blind or shuffled presentation, seeds or sampling rule, dispersion, score math and threshold;
• the complete per-trial receipt fields, not a mutable aggregate;
• a disjoint holdout frozen before tuning, opened once, never reused for iteration, with measurable primary delta, uncertainty rule and protected usefulness, safety or regression guardrails;
• every supplied ceiling or an explicit stricter one, honest terminal states, and each named owner's approval identity, scope, timestamp/evidence and next gate. An experiment win grants no live or durable authority.

A ceiling needs a concrete quantity and unit; a label or promise to predeclare one later is not a ceiling.

Unless the owner supplies another terminal, end plan-only output with STATUS: READY_FOR_OWNER_REVIEW | BLOCKED and EXECUTION: NOT_STARTED. Pending future gates normally mean ready for review. Use BLOCKED only when a missing fact, boundary or decision prevents a valid proposal. Never imply that calibration, trials or approval ran. Any owner-defined exact terminal values, including experiment-result terminals, suppress canonical status and execution.

STRICT_CAP makes the owner's word limit a hard gate. At 250 words or fewer, budget the draft to min(limit - 40, 140) words in at most six nonblank lines. With a counter, compress until the owner limit passes; without one, target min(limit - 50, 130) and never claim an exact count. Print no heading, blank line, internal form, rollback explanation or duplicate terminal. Group only: access; sets/controls; trials/receipt; holdout/decision; limits/stops; gates. Preserve exact numbers, owners and negations; invent no optional mechanism. Treat scope and timing qualifiers, closed field sets and terminal value domains as non-compressible: reuse their literal wording instead of a plausible synonym. When a qualifier applies to each role or item, repeat it for each; collective shorthand is not equivalent. If the owner gives a word range or minimum, its lower bound is also a hard gate and overrides the compact budget. For an exact range, count the final draft with an available whitespace counter and adjust once. Without one, target the range midpoint and tally whitespace-delimited words before sending. For a bare minimum, leave at least 10% headroom. Never exceed a stated maximum. Count exact final text. For a serialized draft, pipe it through the bundled scripts/count_words.py --escaped-newlines; a raw %s count is invalid evidence.

Run CAP_AUDIT before compact output: preserve disjoint one-time holdout and open-once semantics; reproduce an exact receipt field list with no additions; retain immediately when stops are immediate; and when the owner defines exact terminal values, append no canonical status or execution field. If required facts cannot fit, return BLOCKED within the cap rather than omitting them. Scan the complete draft once more: no later duty may contradict a role's only boundary, and every closed value domain retains its literal only or no other qualifier.

Protect evaluator integrity

The evaluator owner and candidate owner must be separate. Candidate write scope excludes the rubric, development/holdout cases, score extractor and acceptance harness; keep them read-only and outside the candidate workspace when possible. A prompt-only prohibition is insufficient when the generator can mutate the score path. Record weaker isolation and do not promote a gameable result.

Before baseline scoring, calibrate the complete path on ordinary positive, known-good, known-bad and tailored deceptive controls. It must rank good above bad, reject deceptive metric gaming, and self-test score extraction and hard failures. Trace every rubric requirement to the frozen request; a hidden judge-only requirement invalidates the evaluator. Missing ground truth or failed calibration is BLOCKED.

Bind every score to an append-only immutable receipt containing: experiment and round ID; candidate ID and target hash; case-set, evaluator, rubric and extractor hashes; judge version/settings; seed or sampling policy and call order; input/data snapshot and configuration; timestamp, repetition ID, environment and budget identity; raw evidence; criterion vector, hard gates, inclusion, retry/error/stop and resulting decision; aggregation formula, inputs, math and aggregate. Content-address it and forbid post-hoc edits. Changed provenance makes results non-comparable: re-baseline instead of carrying a score forward.

Predeclare at least three paired repetitions under identical conditions and compare aggregated gain with measured dispersion. Forbid cherry-picking and favorable retries. Retry only one predeclared transient failure; receipt and count it, never retry to change a result. A calibrated deterministic evaluator may run once when its receipt proves determinism.

Production deployment, credentials, publishing, security or system changes, external contact, spend, destructive actions, durable policy, lessons or wiki writes remain named owner gates. Record the owner, decision and evidence reference before action; experiment success grants none of them.

Run the bounded loop

FRAME -> CALIBRATE -> BASELINE -> GENERATE -> HARD_GATE -> DEV_SCORE -> SELECT -> FINAL_HOLDOUT -> PROMOTE_OR_REVERT

Use development cases for common, edge, adversarial and non-trigger behavior. Prefer one behavioral change per round and three to six genuinely different candidate hypotheses. Run deterministic, public-clean and security gates before judge spend. A hard-gate failure, missing receipt or failed calibration is disqualified, not averaged down.

Score on the unchanged development eval. Select only beyond measured noise; on a near-tie keep the simpler, shorter, lower-risk artifact. One optional graft may combine distinct strengths from a losing candidate, then rerun every gate.

Open the sealed holdout exactly once for the unchanged baseline and champion. Promotion requires all hard gates, the predeclared meaningful primary gain, no protected regression, target-workflow review when subjective, retained baseline and tested rollback. A holdout loss is REVERTED; never tune on that holdout. Any next attempt needs a new holdout, new baseline and new experiment record.

When nested under nobrainer-ultra, update its one canonical TODO after each auditable round and final holdout; do not create a second status owner.

Stop the experiment, not the outcome

Stop the current experiment on its predeclared decision, failed evidence or safety anomaly, configured dry rounds, practical score ceiling, exhausted budget, converged candidates, hard-gate regression, holdout loss, or changed target/constraints/authorization. A null result retains the baseline.

These stops do not close a still-open owner outcome. If proved improvement is still required, diagnose frozen evidence once, choose the smallest high-leverage change and start a fresh bounded experiment. Never reuse a failed holdout or substitute blind retries. Apply Pareto discipline: remove the largest observed failure before polishing lower-impact criteria. Close only when DoD passes or a concrete authority, access, budget or distinct-hypothesis blocker is reported.

Final report

Return winner/baseline identity, development and holdout scores, criterion vectors, hard-gate output, noise/dispersion, calibrated evaluator status, budget, stop reason, diff, rollback and uncertainty. End with:

PROMOTION: PROMOTED | NO_CHANGE | REVERTED | BLOCKED
CHAMPION: <version/ref or BASELINE>
HOLDOUT_RESULT: PASS | FAIL | NOT_RUN
EVALUATOR_STATUS: CALIBRATED | FAILED | UNVERIFIED
SCORE_RECEIPT: <immutable receipt ref or NONE>
NOISE_RESULT: <replications, dispersion and comparable delta or NOT_APPLICABLE>

Use NO_CHANGE when no candidate proves better, REVERTED after holdout loss, and BLOCKED when a gate prevents a valid experiment. Never claim PROMOTED without paired comparable scores and a successful holdout.