# Rseng Notebooks > Covers engineering discipline for computational notebooks: execution-order and hidden-state pitfalls, restart-and-run-all hygiene, keeping notebooks in version control with jupytext, testing notebooks with nbval-style execution checks, parameterizing and batch-running them with papermill, refactoring mature notebook code into importable modules, and deciding what belongs in a notebook versus a package. Use PROACTIVELY when a project contains .ipynb files, when the user works in Jupyter or similar notebooks, mentions notebook reproducibility, testing, version control or parameterization, when a notebook has grown into the de-facto pipeline, or when notebook results must become citable, reviewable artifacts. - Skill: `fdiblen/rseng-notebooks` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add fdiblen/rseng-notebooks` - Raw SKILL.md: https://api.skillmd.com/api/skills/fdiblen/rseng-notebooks/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - License: CC-BY-4.0 - Author: fdiblen (https://skillmd.com/u/fdiblen) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/fdiblen/rseng-notebooks --- # Notebook engineering Notebooks are where most research computing happens - and their superpower (interactive, stateful, out-of-order execution) is exactly their reproducibility hazard. The discipline is not "stop using notebooks"; it is knowing what notebooks are FOR (exploration, narrative, teaching) and applying engineering at the point where a notebook turns into infrastructure without anyone deciding it should. The CodeRefinery Jupyter lesson teaches the same split. ## Hidden state: the core hazard A notebook's on-screen code and its kernel's actual state can disagree: cells run out of order, deleted cells leave live variables, and a figure can come from code that no longer exists. Working rules: - Restart-and-run-all IS the test: before trusting, committing or sharing any notebook, restart the kernel and run top to bottom. A notebook that only works in one execution order is broken - fix the order now, while you still remember it. - Keep cells re-runnable: no cell should break when run twice (guard appends, recreations, cumulative mutations); avoid mutating a variable across distant cells - derive new names instead. - Definitions up top: imports, parameters and configuration in the first cells; a parameters cell makes intent explicit and enables parameterization later. - Watch execution counts: out-of-sequence numbers in a "final" notebook are the tell (rseng-version-control-review reviewers can see them too). ## Version control that works Raw .ipynb diffs are JSON noise with embedded outputs: - Pair notebooks with a script/markdown twin via jupytext: the text twin is the reviewable, diffable, mergeable source; the .ipynb carries outputs locally. - Decide the output policy explicitly: strip outputs before commit (clean diffs, smaller repos - a pre-commit hook automates it) OR commit executed notebooks deliberately as result records (rseng-reproducibility's compendium may want exactly that); mixing the two policies in one repo confuses everyone. - Large embedded outputs (images, tables) are the notebook form of committed data - the same instincts apply (rseng-data-management). ## Testing and automation - Execution tests: run notebooks in CI and fail on errors (nbval-style pytest integration checks stored outputs too; plain execution via papermill or nbconvert catches rot). A notebook that no longer runs is the most common form of repository decay (rseng-ci-cd; budget-conscious: run on reduced data - rseng-green-computing). - Parameterize with papermill: a tagged parameters cell turns a notebook into a callable with arguments - the honest bridge from "notebook" to "batch job" (sweeps via rseng-hpc-computing job arrays or rseng-workflows without abandoning the notebook form). - Environments: a notebook without a pinned environment is a screenshot; keep the kernel's environment in the repository's lockfile (rseng-reproducible-environments) and Binder-ready when sharing is the goal (rseng-reproducibility). - Seeds and data paths follow the same rules as any code (rseng-defensive-coding) - notebooks get no exemption from provenance. ## The graduation path Notebooks accumulate load-bearing code; the move is extraction, not rewrite: - When a function is used twice, or another notebook copies it, extract it to a module in the repository and import it - the notebook keeps the narrative, the module gets tests (rseng-testing) and design (rseng-software-design). Autoreload during development keeps the loop tight. - When a notebook IS the pipeline (runs nightly, feeds a paper), graduate the computation to scripts/workflow stages (rseng-workflows) and keep a thin notebook as the exploratory window onto results. - The notebook that remains is the good kind: narrative analysis, figures with their reasoning, teaching material (rseng-trainer) - prose and code interleaved doing what neither does alone (rseng-science-communication for the storytelling layer). ## Working with this skill This skill is source-independent: its authority is the Jupyter ecosystem tooling and the CodeRefinery lesson linked below. Learn more (verified): - https://jupyter.org - Project Jupyter - https://jupytext.readthedocs.io - jupytext notebook/text pairing - https://github.com/computationalmodelling/nbval - nbval notebook validation - https://papermill.readthedocs.io - papermill parameterization - https://coderefinery.github.io/jupyter/ - CodeRefinery Jupyter lesson ## Related skills Check whether any of these applies before moving on: - rseng-hpc-computing - papermill batch sweeps on clusters - rseng-reproducibility - executed notebooks as deliberate result records - rseng-science-communication - narrative layer of analysis notebooks - rseng-scientific-visualization - figure-producing notebook cells - rseng-trainer - notebooks as teaching material - rseng-version-control-review - jupytext twins make diffs reviewable