# Nb Run

> Run a Jupyter notebook headless on a remote machine through nbconvert with a named profile of env flags: pre-flight check, launch detached from ssh, notification when it finishes, failure triage and comparison against the previous run. Use when asked to "run this notebook", "run the experiment", "rerun with different parameters", "train on the server", "run the smoke/cheap variant", "watch the run", "tell me when it's done", "why did the notebook fail", "what's the status of the run", and when reproducing a run from an experiment write-up. Not for editing notebooks interactively and not for triggering Airflow DAGs.

- Skill: `viderstar/nb-run` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add viderstar/nb-run`
- Raw SKILL.md: https://api.skillmd.com/api/skills/viderstar/nb-run/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ViderStar (https://skillmd.com/u/viderstar)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/viderstar/nb-run

---


# Run a notebook with a profile

A heavy notebook runs for hours, each run wants its own set of environment flags, and a
crash is normally discovered only when you sit back down at the computer. This skill
removes both problems: a named profile instead of flags held in your head, and a signal
on completion instead of "I'll go check on it".

Entry point is `scripts/nb_run.py`, standard library only. In Claude Code the absolute
path is `${CLAUDE_PLUGIN_ROOT}/skills/nb-run/scripts/nb_run.py`; otherwise build it from
the directory this SKILL.md was read from.

`jupyter` is needed where the notebook *executes*, not where this script runs — on a
laptop it is usually absent, so **default to the server**. The notebook does not even
have to exist locally: a remote run needs only a file name and a directory.

## 1. One-time setup

```bash
nb_run.py servers add gpu --host 10.0.0.5 --user me \
    --workdir /home/me/research --jupyter /home/me/venv/bin/jupyter --default
nb_run.py servers ssh-config     # optional: write them into ~/.ssh/config
nb_run.py servers check gpu
```

`servers add` writes `~/.config/nb-run/servers.json`. `--jupyter` matters more than it
looks: a system `jupyter` often belongs to a different Python and lacks the kernel the
notebook declares. `--docker CONTAINER` covers the common case where jupyter lives in a
per-user container rather than on the host — the run is then wrapped in `docker exec`
automatically, while the logs still land on the host so they are readable without docker.

A machine that already has an alias in `~/.ssh/config` needs none of this: pass
`--remote <alias> --remote-dir <dir>`.

If a machine answers `Permission denied`, your key is not there: `ssh-copy-id <host>` —
**the human runs that**, it asks for a password and a password must not travel through
an agent or a chat. A key with a passphrase has to be in the agent, since this tool
connects with `BatchMode=yes` and cannot answer a prompt.

**Bastions** are described by `--proxy-jump` (or a `ProxyJump` line in `~/.ssh/config`).
This tool calls plain `ssh <host>` and picks the config up by itself.

## 2. Profiles

They live in `nb-run.json` next to the notebooks or anywhere above it in the tree. A
fully annotated sample is `assets/nb-run.json.example`.

```json
{
  "remote": { "host": "gpu", "dir": "/home/me/research" },
  "notify": ["mac"],
  "profiles": {
    "smoke": { "description": "cheap sanity check (minutes)",
               "env": { "RUN_TUNING": "0", "TRIALS": "2" } },
    "full":  { "description": "full run (~3.5 h)",
               "env": { "RUN_TUNING": "1", "TRIALS": "40" } }
  }
}
```

Host resolution order: `--remote` → `remote.host` in `nb-run.json` → `defaults.server`
in `servers.json`. Directory: `--remote-dir` → `remote_dir` in the profile →
`remote.dir` → `run.workdir` of the machine.

No profile yet? Build one from the parameters the human gave you, show the resulting
JSON, and only then run.

## 3. Commands

| Command | What it does |
|---|---|
| `nb_run.py list <nb>` | profiles visible to this notebook, and recent runs |
| `nb_run.py check <nb> -p full` | **pre-flight** — see below, always before a long run |
| `nb_run.py prepare --branch X` | on the server: `git pull --ff-only`, dependency sync |
| `nb_run.py run <nb> -p full` | launch; prints `RUN_ID` immediately and returns |
| `nb_run.py wait <id> --daemon` | wait for the end and notify, without holding the session |
| `nb_run.py status <id>` | running / failed / done, no waiting |
| `nb_run.py report <id>` | verdict, duration delta and metric delta vs the previous run |

`run` starts the notebook under `nohup` on the server and **detaches it from ssh**: a
closed laptop, sleep or a dropped VPN no longer kill it.

## 4. How to work

1. **`check` before anything long.** It catches what otherwise costs the full three
   hours: a misspelled flag (present in the profile, never read by the notebook), a
   missing kernel, a dirty working tree on the server, a cache-reuse profile with no
   refresh flags, a busy GPU.
2. **`--dry-run` when anything is unusual** — it prints the env and the exact script
   that would be shipped.
3. **`run`** returns a `RUN_ID` in a second. Then either:
   - `wait <id> --daemon --notify mac` — the wait forks into the background and the
     notification arrives even after this session ends. Do this for multi-hour runs.
   - watch the log directly if the human is sitting there right now:
     ```
     ssh <host> tail -f <LOG> | grep -E --line-buffered "epoch|step|ndcg|Traceback|Error|FAILED|Killed|OOM|CUDA out of memory"
     ```
     A filter with only success markers goes silent on a crash — and silence is
     indistinguishable from "still computing". Always include the error branch.
4. **When it ends — `report`.** It pulls metrics out of the log, compares them with the
   last successful run of the same profile, and shows the place it died rather than the
   last twenty lines.
5. **If it failed, do not just rerun it.** Show the traceback and propose what to change.

## 5. Notifications

**Sound is on by default and differs by outcome:** failure `Basso`, success `Glass`. A
silent banner after a three-hour run gets missed, so the sound is not decoration.

| Channel | When it works | Setup |
|---|---|---|
| `mac` | while the laptop is awake | nothing, works out of the box |
| `webhook` | always, sent by the server itself | `NOTIFY_WEBHOOK_URL` (Slack-shaped incoming webhook) |
| `telegram` | always | `NOTIFY_TG_TOKEN`, `NOTIFY_TG_CHAT_ID` |

Settings live in `~/.config/nb-run/notify.env` (chmod 600, outside git). Tuning:
`NOTIFY_SOUND_FAIL`, `NOTIFY_SOUND_OK`, `NOTIFY_SOUND=off`, `NOTIFY_SPEAK=1`. Check with
`python3 notify.py --list`; test the sound with `python3 notify.py --fail "test" "..."`.

A webhook URL is itself a secret — whoever has it can post to that channel. It is never
shipped from the laptop to the server; to get notifications from the server while the
laptop is shut, the human puts `notify.env` on the server themselves.

## 6. Traps worth knowing

- **A cache can silently replace your result.** A trial that returned a cached number
  instead of an independent run destroys any estimate of variance. If the task is to
  measure spread or compare configurations, set the refresh flags and say so out loud.
  `check` warns when a profile is named `*fresh*` but no cache flag is on.
- **A control arm has to be retrained, not copied.** Reusing a checkpoint for the control
  invalidates the comparison.
- **`--inplace` overwrites the notebook — on the server.** That is where the working tree
  must be clean, not on the laptop. `check` verifies this.
- **A run that vanished without status.json** is almost always OOM or a reboot. `wait`
  recognises that case and says so instead of waiting forever.
- **Outputs go into the .ipynb, not the log,** when a cell renders instead of printing.
  Either print the numbers to stdout or fetch the executed notebook with
  `wait <id> --fetch-notebook`.