Notebook Refactor

Notebook Refactoring Skill

Convert Jupyter notebook exploration code into clean, tested Python modules.

Process

1. Analyze the Notebook

Read the notebook and identify:

• Data loading cells -> src/data/ module
• Preprocessing/transformation cells -> src/data/ or src/preprocessing/ module
• Model definition cells -> src/models/ module
• Training loop cells -> src/training/ module
• Evaluation/metrics cells -> src/evaluation/ module
• Visualization cells -> keep in notebook (these are exploratory)
• Configuration values (magic numbers, paths) -> src/config/ or config file

2. Extract Functions

For each group of cells:

1. Identify inputs and outputs of the cell block
2. Extract into a function with:
   • Type-annotated parameters for all inputs
   • A clear return type
   • A docstring explaining what it does and why
3. Replace hardcoded values with parameters
4. Remove display(), print() debugging statements
5. Keep the notebook cell but replace the code with an import + function call

3. Write Tests

For each extracted function, write tests that:

• Use small, deterministic test fixtures (not the full dataset)
• Test the function's contract (input types -> output types/shapes)
• Test edge cases (empty input, single row, missing values)
• Use np.testing.assert_allclose for numerical outputs
• Do NOT test exact numerical values from model operations (non-deterministic)

4. Update the Notebook

After extraction, the notebook should:

• Import from the new modules instead of defining functions inline
• Still be runnable end-to-end
• Serve as a high-level walkthrough / documentation of the pipeline
• Keep exploratory visualizations and analysis inline

5. Refactoring Checklist

Before marking complete:

• All extracted functions have type hints
• All extracted functions have tests
• Notebook still runs end-to-end with imports
• No hardcoded paths or magic numbers remain
• No unused imports in extracted modules
• pyproject.toml or setup.py updated if new packages are needed