Notebook Debug

Autonomous Notebook Debug & Fix

You are debugging the notebook at $ARGUMENTS. Run it end-to-end, collect all errors, fix them, and re-run until all cells pass. Work autonomously — don't ask the user unless you're genuinely stuck on a design decision.

Safety Rules

You may freely read, search, and explore any files needed to understand the notebook's dependencies and data. However, ask the user for confirmation before:

• Installing packages or modifying the conda environment
• Submitting SLURM jobs
• Deleting or overwriting data files

When in doubt, ask. The cost of pausing is low; the cost of clobbering data is high.

Environment

• Python: use the conda env sobulk24 at /gpfs/commons/home/tbotella/miniconda3/envs/sobulk24/bin/python
• Papermill (always available): papermill input.ipynb /dev/null --log-output 2>&1 | tee run.log
• Working dir: cd to the notebook's directory before running.

Workflow

Phase 1 — Initial Run

1. Syntax check first (catches literal newlines in f-strings, unmatched parens):

   python3 -c "import json,ast; [ast.parse(''.join(c['source'])) for c in json.load(open('NB.ipynb'))['cells'] if c['cell_type']=='code']"
   

   Fix any syntax errors before running.

2. Run the notebook with nohup (survives SSH disconnects):

   cd /path/to/notebook/dir
   nohup /gpfs/commons/home/tbotella/miniconda3/envs/sobulk24/bin/python -m papermill \
     notebook.ipynb /dev/null --log-output > run_stdout.txt 2>&1 &
   echo "PID: $!"
   

3. Monitor progress — check tail -20 run_stdout.txt periodically. Don't poll in a tight loop; wait 60-120s between checks for long-running notebooks.

4. Collect all errors once the run completes:

   grep -E "ERROR:|FAILED|OK \(|^\[" run_stdout.txt
   

Phase 2 — Diagnose & Fix

5. Classify errors: distinguish root-cause bugs from cascade failures. A cascade failure is when cell N fails because cell M (earlier) failed and didn't define a variable. Fix only the root cause — cascades resolve automatically.

6. Edit .ipynb cells — use one of:

   • NotebookEdit tool (preferred for small edits): find cell index via Grep on the .ipynb JSON.
   • Python JSON manipulation via Bash (for complex multi-line edits):

     import json
     nb = json.load(open('notebook.ipynb'))
     cell = nb['cells'][INDEX]
     src = ''.join(cell['source'])
     # ... modify src ...
     cell['source'] = [new_src]
     json.dump(nb, open('notebook.ipynb', 'w'), indent=1)
     

   • Never use the Edit tool on .ipynb files — it doesn't understand JSON cell structure.

7. Common bug patterns:

   • Cache loading path missing variables: if a cell has if cache: load... else: compute..., ensure ALL variables defined in the else branch are also set in the if branch.
   • Literal newlines in f-strings: JSON source arrays split strings across lines. ast.parse() catches these. Fix by merging the JSON lines into one with \n escapes.
   • Duplicate column on merge: if DataFrame A already has column X from a previous merge, merging again with column X creates X_x and X_y. Check before merging: df_merged = df if 'col' in df.columns else df.merge(...).
   • O(n^2) membership tests: [x for x in big_list if x in big_array] where big_array is a numpy array → use set() or np.isin() or BLAS matrix multiply.
   • Sample count mismatch: tools that process all files in a directory may include QC-failed samples. Use os.listdir() for the full list, then inner merge with QC-passed samples to filter.

8. Syntax check again after all fixes (step 1).

Phase 3 — Verify

9. Re-run the notebook (same nohup pattern as step 2).

10. Confirm all cells pass: grep "FAILED\|ERROR" run_stdout.txt should return nothing. If errors remain, go back to Phase 2. Limit to 3 iterations — if still failing after 3, report remaining errors to the user.

Performance Rules

• Cache slow cells (>5min): wrap in if cache_valid: load else: compute; save. Use a key file with sorted sample list for invalidation. Helpers: _cache_ok(name, sdf) checks key; _cache_save_key(name, sdf) writes key.
• Never reload data already in memory: if cell N loaded a large file, cell M should reuse the variable, not re-read the file.
• Prefer vectorized numpy/BLAS over Python loops: for large matrix operations, mask.astype(float32) @ matrix is orders of magnitude faster than Python indexing loops.
• Use polars for file I/O: pl.read_csv(..., n_threads=1) in a ThreadPoolExecutor for parallel GPFS reads when loading many files.

Output

When done, provide a concise summary:

• Total cells: X, Passed: Y, Fixed: Z
• List of fixes applied (one line each)
• Total runtime
• Any warnings or known issues remaining