Nb Run

Run a notebook with a profile

A heavy notebook runs for hours, each run wants its own set of environment flags, and a crash is normally discovered only when you sit back down at the computer. This skill removes both problems: a named profile instead of flags held in your head, and a signal on completion instead of "I'll go check on it".

Entry point is scripts/nb_run.py, standard library only. In Claude Code the absolute path is ${CLAUDE_PLUGIN_ROOT}/skills/nb-run/scripts/nb_run.py; otherwise build it from the directory this SKILL.md was read from.

jupyter is needed where the notebook executes, not where this script runs — on a laptop it is usually absent, so default to the server. The notebook does not even have to exist locally: a remote run needs only a file name and a directory.

1. One-time setup

nb_run.py servers add gpu --host 10.0.0.5 --user me \
    --workdir /home/me/research --jupyter /home/me/venv/bin/jupyter --default
nb_run.py servers ssh-config     # optional: write them into ~/.ssh/config
nb_run.py servers check gpu

servers add writes ~/.config/nb-run/servers.json. --jupyter matters more than it looks: a system jupyter often belongs to a different Python and lacks the kernel the notebook declares. --docker CONTAINER covers the common case where jupyter lives in a per-user container rather than on the host — the run is then wrapped in docker exec automatically, while the logs still land on the host so they are readable without docker.

A machine that already has an alias in ~/.ssh/config needs none of this: pass --remote <alias> --remote-dir <dir>.

If a machine answers Permission denied, your key is not there: ssh-copy-id <host> — the human runs that, it asks for a password and a password must not travel through an agent or a chat. A key with a passphrase has to be in the agent, since this tool connects with BatchMode=yes and cannot answer a prompt.

Bastions are described by --proxy-jump (or a ProxyJump line in ~/.ssh/config). This tool calls plain ssh <host> and picks the config up by itself.

2. Profiles

They live in nb-run.json next to the notebooks or anywhere above it in the tree. A fully annotated sample is assets/nb-run.json.example.

{
  "remote": { "host": "gpu", "dir": "/home/me/research" },
  "notify": ["mac"],
  "profiles": {
    "smoke": { "description": "cheap sanity check (minutes)",
               "env": { "RUN_TUNING": "0", "TRIALS": "2" } },
    "full":  { "description": "full run (~3.5 h)",
               "env": { "RUN_TUNING": "1", "TRIALS": "40" } }
  }
}

Host resolution order: --remote → remote.host in nb-run.json → defaults.server in servers.json. Directory: --remote-dir → remote_dir in the profile → remote.dir → run.workdir of the machine.

No profile yet? Build one from the parameters the human gave you, show the resulting JSON, and only then run.

3. Commands

run starts the notebook under nohup on the server and detaches it from ssh: a closed laptop, sleep or a dropped VPN no longer kill it.

4. How to work

1. check before anything long. It catches what otherwise costs the full three hours: a misspelled flag (present in the profile, never read by the notebook), a missing kernel, a dirty working tree on the server, a cache-reuse profile with no refresh flags, a busy GPU.
2. --dry-run when anything is unusual — it prints the env and the exact script that would be shipped.
3. run returns a RUN_ID in a second. Then either:
   • wait <id> --daemon --notify mac — the wait forks into the background and the notification arrives even after this session ends. Do this for multi-hour runs.
   • watch the log directly if the human is sitting there right now:

     ssh <host> tail -f <LOG> | grep -E --line-buffered "epoch|step|ndcg|Traceback|Error|FAILED|Killed|OOM|CUDA out of memory"
     

     A filter with only success markers goes silent on a crash — and silence is indistinguishable from "still computing". Always include the error branch.
4. When it ends — report. It pulls metrics out of the log, compares them with the last successful run of the same profile, and shows the place it died rather than the last twenty lines.
5. If it failed, do not just rerun it. Show the traceback and propose what to change.

5. Notifications

Sound is on by default and differs by outcome: failure Basso, success Glass. A silent banner after a three-hour run gets missed, so the sound is not decoration.

Settings live in ~/.config/nb-run/notify.env (chmod 600, outside git). Tuning: NOTIFY_SOUND_FAIL, NOTIFY_SOUND_OK, NOTIFY_SOUND=off, NOTIFY_SPEAK=1. Check with python3 notify.py --list; test the sound with python3 notify.py --fail "test" "...".

A webhook URL is itself a secret — whoever has it can post to that channel. It is never shipped from the laptop to the server; to get notifications from the server while the laptop is shut, the human puts notify.env on the server themselves.

6. Traps worth knowing

• A cache can silently replace your result. A trial that returned a cached number instead of an independent run destroys any estimate of variance. If the task is to measure spread or compare configurations, set the refresh flags and say so out loud. check warns when a profile is named *fresh* but no cache flag is on.
• A control arm has to be retrained, not copied. Reusing a checkpoint for the control invalidates the comparison.
• --inplace overwrites the notebook — on the server. That is where the working tree must be clean, not on the laptop. check verifies this.
• A run that vanished without status.json is almost always OOM or a reboot. wait recognises that case and says so instead of waiting forever.
• Outputs go into the .ipynb, not the log, when a cell renders instead of printing. Either print the numbers to stdout or fetch the executed notebook with wait <id> --fetch-notebook.