Autoresearch
A study turns "make this metric better" into a loop that runs for hours or days without losing the thread. You own the science: which ideas to try, how to implement them, what a result means. OpenScience owns the clock: it tracks every metric the script logs, notices when a run ends, kills runs that break the study's criteria, keeps the budget, renders the ledger, and wakes this session with a "Study update" whenever there is news.
Before the study
- Agree the objective with the user: one metric, its direction, the budget, the compute target, how many runs may be live at once, and the kill criteria. The budget is a spend decision and belongs to this study alone: never carry one over from an earlier study or an earlier instruction in the session. If the request names no budget, ask once, with a recommendation (a time budget such as 2 hours with a per-run kill rule, or a target value), even under an autonomous setting. Everything else, infer or default.
- Read the code and data first. A study needs a training or analysis script the harness can run repeatedly with different configuration, and a fixed evaluation that computes the metric. The evaluation does not change once the baseline has run.
- Create the study with
study create. Propose the baseline with priority 1000 and at least three first ideas withstudy propose, of different kinds (a different component, objective, data treatment or search strategy each), not three magnitudes of one knob. An idea has a title, what it changes, why it should help, an expected improvement in metric units times your confidence, and the configuration it needs; a configuration already in the study is rejected. The hypotheses skill is the tool for turning a vague direction into ideas worth queueing. - If the review gate is on (the default), get a read-only critique of the training and
evaluation code before the baseline runs: load the peer-review skill and read it
yourself, or delegate one read to an
exploreworker (Task tool,subagent_type: "explore") with the file paths and the angle (leakage, metric definition, split hygiene). Fix anything marked blocking. A baseline built on a leaking split or a misspelled metric wastes every run after it.
Every run
- The script imports
openscience_track(orwandb, which is shimmed) and logs the study metric at every evaluation plus anything worth a curve:track.log({"val_loss": v, "lr": lr}, step=step). It setstrack.summary["val_loss"]to the final value and callstrack.finish().openscience_trackis OpenScience's own tracking module, written into the study root under.openscience/sdk/and added to the run's PYTHONPATH and uploads bystudy start; it is not on PyPI or GitHub, so do not search for it or ask a worker to audit it. Its whole API isinit,log,summary,config,finish. - Start exactly one run per idea with
study start, passing the command and the configuration the idea needs. Never start a second run for the same idea; propose a new idea if a variant is worth trying. - The run executes in the study root: keep the script, its data (copy it in) and its
outputs there, write the command as
python train.py ..., and nameuploadsandartifactsrelative to the root. The tracking SDK travels with every remote run on its own; the study's ledger files never do. A remote study is approved once, when it is created; its runs then dispatch without a card each. - Keep up to the study's concurrency live, and keep at least three ideas queued so a free slot never waits on you; propose in batches when the queue thins.
- When runs finish within a few minutes, stay in the turn: a run that settles within the
dispatch's grace comes back with its outcome in that step, otherwise one
compute_job waitfor it; record it, start the next. Wake-ups are for runs that outlast a turn. While a long run is live, implement the next idea rather than idling. - When a "Study update" reports a run ended, read its numbers with
experiments compare(orexperiments serieswhen the curve matters), decide keep or revert against the baseline and the best, and record the verdict withstudy record: the analysis, a conclusion, and any lesson that should shape later ideas. Mark the first reference run withbaseline: true. - A killed run is data, not an error: record why it diverged and what that rules out.
Steering
The user can add a directive from the Autoresearch pane or in chat while the study runs. A directive is a standing rule for the rest of the study (it appears in your study reminder): re-rank the queue and change the next run to honour it, and say what changed. Every sixth run, and whenever four runs in a row fail to beat the best, step back: re-read the lessons, name the kinds of change tried, drop near-duplicates, and change the kind of idea rather than its magnitude.
Judgement
- Rank by expected value, but keep a few cheap, orthogonal ideas in the queue so a plateau does not stall the study.
- Do not repeat a configuration that already has a run; check
study statusbefore proposing. - Prefer changes to the training script over changes to the evaluation. If the evaluation is wrong, stop and say so.
- Ask only when input or authority is missing. Do not ask whether to continue while budget remains.
- When the budget or target is reached, or the queue is empty and no idea is worth its
cost, conclude with
study conclude: what was learned, the best configuration with its metric, and what remains open. The ledger files (study.md,ideas.md,results.tsv,lessons.md) in the working folder are the record; the Autoresearch pane shows the same data live.
Writing up
For a paper or report, load ml-paper-writing or paper-writing and build from the ledger and the tracked runs: the baseline, the best configuration, the ablations that changed the metric, and the figures the data supports (figures skill). Kept runs are claims with evidence; reverted runs are the ablations that make the claims honest.
