SWEStash Notebook to Production
Turn exploratory notebooks into code that runs unattended, produces the same result twice, and fails loudly. The refactor is incremental — the notebook keeps working at every step — never a from-scratch rewrite.
Workflow
1. Triage — not every notebook earns this
Productionize by re-run frequency × business impact. A notebook someone
re-runs by hand every quarter is the first candidate; a one-off exploration is
not — archive it reproducibly instead: the query or a data snapshot, a
pinned environment (pip freeze at minimum), and a one-line header (question,
finding, date, author). "Archived" without the data recipe and environment is
deleted-with-extra-steps.
For a backlog of many notebooks, also: extract logic duplicated across them into a shared utility package (one fix, not N), and set a lightweight convention for future notebooks likely to graduate — imports at top, parameters in the first cell, logic in functions — so the next migration is cheap.
2. Extract logic into modules
Move transformations out of cells into importable, typed functions in a real
package (src/), leaving the notebook as a thin driver that imports and
calls. Separate I/O (load/save) from transformation logic — pure
transformation functions are what you can test. Restartability check: does it
run top-to-bottom in a fresh kernel? Hidden execution-order state is the first
bug this flushes out.
3. Parameterize everything hardcoded
Dates, paths, connection strings, thresholds → function arguments driven by a config file or CLI args (or papermill parameters if the notebook itself stays the execution vehicle). The run for "last month" and the backfill for "March 2024" must be the same code with different parameters.
4. Test the transformations
The extracted pure functions get unit tests: known-input → known-output cases, plus edge cases the data will eventually throw (empty frame, nulls, duplicate keys, timezone boundaries). A small golden sample of real data as a fixture catches most regressions. No tests, no schedule — an untested unattended job is a silent-corruption machine.
5. Pin the environment
Reproducibility means the same code AND the same dependencies: a lock file
(uv lock, poetry.lock, pip-compile) or a container image. Pin the data
too where possible — snapshot inputs or record query + as-of timestamp, so a
rerun of last month's report produces last month's numbers.
6. Schedule and operate
Size the scheduler to the team: cron or a CI schedule (GitHub Actions) first; an orchestrator (Airflow/Dagster) only when there are real DAG dependencies across jobs. Unattended means operated: fail loudly (alert on failure AND on suspicious output — empty result, row count off), log enough to debug a 6am failure, and make the job idempotent so a retry can't double-append or double-send.
Principles
- Incremental (KISS): extract one section at a time; the notebook keeps running throughout.
- YAGNI: no orchestrator, feature store, or microservice for a weekly report — cron + a tested module is production.
- SRP: load, transform, and render are separate functions with separate tests.
Cross-skill boundaries
- The notebook trains a model (feature engineering, training, evaluation, retraining schedule) → ml-pipeline-design — training pipelines have their own stage/validation/gating design.
- Recurring ELT into a warehouse feeding many consumers → data-pipeline-design.
- Still figuring out what the analysis should say → exploratory-data-analysis.
- CI mechanics for the scheduled run → cicd-pipeline; test design depth → test-suite-design.