# Notebook Debug

> Autonomously run a Jupyter notebook, collect all errors, fix them iteratively, and re-run until clean. Use when asked to debug, fix, or test a notebook.

- Skill: `theob0t/notebook-debug` (Agent Skill)
- Install (CLI): `npx skillmds@latest add theob0t/notebook-debug`
- Raw SKILL.md: https://api.skillmd.com/api/skills/theob0t/notebook-debug/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: Theob0t (https://skillmd.com/u/theob0t)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/theob0t/notebook-debug

---


# Autonomous Notebook Debug & Fix

You are debugging the notebook at `$ARGUMENTS`. Run it end-to-end, collect all errors,
fix them, and re-run until all cells pass. Work autonomously — don't ask the user unless
you're genuinely stuck on a design decision.

## Safety Rules

You may freely read, search, and explore any files needed to understand the notebook's
dependencies and data. However, **ask the user for confirmation before:**
- Installing packages or modifying the conda environment
- Submitting SLURM jobs
- Deleting or overwriting data files

When in doubt, ask. The cost of pausing is low; the cost of clobbering data is high.

## Environment

- **Python**: use the conda env `sobulk24` at `/gpfs/commons/home/tbotella/miniconda3/envs/sobulk24/bin/python`
- **Papermill** (always available): `papermill input.ipynb /dev/null --log-output 2>&1 | tee run.log`
- **Working dir**: `cd` to the notebook's directory before running.

## Workflow

### Phase 1 — Initial Run

1. **Syntax check first** (catches literal newlines in f-strings, unmatched parens):
   ```bash
   python3 -c "import json,ast; [ast.parse(''.join(c['source'])) for c in json.load(open('NB.ipynb'))['cells'] if c['cell_type']=='code']"
   ```
   Fix any syntax errors before running.

2. **Run the notebook with nohup** (survives SSH disconnects):
   ```bash
   cd /path/to/notebook/dir
   nohup /gpfs/commons/home/tbotella/miniconda3/envs/sobulk24/bin/python -m papermill \
     notebook.ipynb /dev/null --log-output > run_stdout.txt 2>&1 &
   echo "PID: $!"
   ```

3. **Monitor progress** — check `tail -20 run_stdout.txt` periodically.
   Don't poll in a tight loop; wait 60-120s between checks for long-running notebooks.

4. **Collect all errors** once the run completes:
   ```bash
   grep -E "ERROR:|FAILED|OK \(|^\[" run_stdout.txt
   ```

### Phase 2 — Diagnose & Fix

5. **Classify errors**: distinguish root-cause bugs from cascade failures.
   A cascade failure is when cell N fails because cell M (earlier) failed and didn't define
   a variable. Fix only the root cause — cascades resolve automatically.

6. **Edit .ipynb cells** — use one of:
   - **NotebookEdit tool** (preferred for small edits): find cell index via `Grep` on the .ipynb JSON.
   - **Python JSON manipulation via Bash** (for complex multi-line edits):
     ```python
     import json
     nb = json.load(open('notebook.ipynb'))
     cell = nb['cells'][INDEX]
     src = ''.join(cell['source'])
     # ... modify src ...
     cell['source'] = [new_src]
     json.dump(nb, open('notebook.ipynb', 'w'), indent=1)
     ```
   - **Never use the Edit tool on .ipynb files** — it doesn't understand JSON cell structure.

7. **Common bug patterns**:
   - **Cache loading path missing variables**: if a cell has `if cache: load... else: compute...`,
     ensure ALL variables defined in the `else` branch are also set in the `if` branch.
   - **Literal newlines in f-strings**: JSON source arrays split strings across lines.
     `ast.parse()` catches these. Fix by merging the JSON lines into one with `\n` escapes.
   - **Duplicate column on merge**: if DataFrame A already has column X from a previous merge,
     merging again with column X creates X_x and X_y. Check before merging:
     `df_merged = df if 'col' in df.columns else df.merge(...)`.
   - **O(n^2) membership tests**: `[x for x in big_list if x in big_array]` where
     `big_array` is a numpy array → use `set()` or `np.isin()` or BLAS matrix multiply.
   - **Sample count mismatch**: tools that process all files in a directory may include
     QC-failed samples. Use `os.listdir()` for the full list, then `inner` merge with
     QC-passed samples to filter.

8. **Syntax check again** after all fixes (step 1).

### Phase 3 — Verify

9. **Re-run the notebook** (same nohup pattern as step 2).

10. **Confirm all cells pass**: `grep "FAILED\|ERROR" run_stdout.txt` should return nothing.
    If errors remain, go back to Phase 2. Limit to 3 iterations — if still failing after 3,
    report remaining errors to the user.

## Performance Rules

- **Cache slow cells (>5min)**: wrap in `if cache_valid: load else: compute; save`.
  Use a key file with sorted sample list for invalidation.
  Helpers: `_cache_ok(name, sdf)` checks key; `_cache_save_key(name, sdf)` writes key.
- **Never reload data already in memory**: if cell N loaded a large file, cell M should
  reuse the variable, not re-read the file.
- **Prefer vectorized numpy/BLAS over Python loops**: for large matrix operations,
  `mask.astype(float32) @ matrix` is orders of magnitude faster than Python indexing loops.
- **Use polars for file I/O**: `pl.read_csv(..., n_threads=1)` in a ThreadPoolExecutor
  for parallel GPFS reads when loading many files.

## Output

When done, provide a concise summary:
- Total cells: X, Passed: Y, Fixed: Z
- List of fixes applied (one line each)
- Total runtime
- Any warnings or known issues remaining