Notebooks

Notebooks

A single skill for authoring, validating, and delivering reproducible analysis notebooks. Marimo is the default format; Jupyter is supported for existing .ipynb files and when a downstream tool requires JSON. Conversion between the two formats is part of this skill.

A notebook is not "done" until it has been executed end-to-end on a fresh kernel and every figure is embedded in the delivered file.

Instructions

1. Pick the format.

   • New notebook: write a marimo .py notebook. Use the canonical cell layout (one concept per cell, final expression renders, no if guards around outputs, no try/except for control flow).
   • Existing .ipynb to extend or polish: keep it as Jupyter unless the user asks to convert.
   • Conversion: see "Convert between marimo and Jupyter" below.

2. Outline before coding. Write the notebook plan (purpose, data sources, analysis steps, expected outputs/plots) as the first markdown cell, then implement against that plan.

3. Keep marimo cells clean. These are hard rules for every .py notebook:

   • Markdown cells use one plain triple-quoted string: mo.md(r"""...""") or mo.md(f"""...""") only when interpolation is required. Put the prose directly inside the string; never paste quoted string fragments such as " ... " lines inside the markdown body.
   • Do not leave empty generated cells, whitespace-only cells, or @app.cell def _(): return placeholders. Remove them before final verification.
   • Do not accept a marimo "fix" prompt blindly. If one is accepted during interactive editing, inspect the diff immediately and remove unintended PEP 723/header/cell churn.

4. Set up the kernel and dependencies.

   • Marimo. Pin dependencies in the PEP 723 script header at the top of the .py file:

     # /// script
     # requires-python = ">=3.12"
     # dependencies = [
     #     "marimo",
     #     "polars",
     #     "duckdb",
     #     "matplotlib",
     #     # ... add every import used in the notebook
     # ]
     # ///
     

     Run with uvx marimo run --sandbox <notebook.py> or edit interactively with uvx marimo edit --sandbox <notebook.py>. The sandbox reads the header and resolves the notebook environment.
   • Jupyter. Register a named ipykernel for the project's pixi env before the first execution and pin the kernel in the notebook metadata. The kernel name is mandatory — the generic python3 kernel leaks the system interpreter:

     pixi run python -m ipykernel install --user --name <project> --display-name "<project> (pixi)"
     

     Then in <notebook>.ipynb confirm:

     "kernelspec": {"name": "<project>", "display_name": "<project> (pixi)"}
     

     Add every import used in the notebook to pixi.toml so the kernel can resolve it from a clean install.

5. Load data with project-relative paths. Prefer DuckDB for TSV/Parquet (duckdb.read_csv, duckdb.read_parquet). Avoid absolute paths and ~. Avoid hidden state from the runtime working directory.

6. Run checks and all cells to generate plots. Execute the notebook headlessly on a fresh kernel before delivery:

   • Marimo: run uvx marimo check --strict <notebook.py> before export. Then run uvx marimo export ipynb <notebook.py> -o <notebook.executed.ipynb> --include-outputs --sandbox -f. For a deterministic HTML artifact, use uvx marimo export html <notebook.py> -o <notebook.html> --sandbox -f. Run the strict check again after the final edit/export cycle.
   • Jupyter: resolve the installed skill root, then run its self-provisioning helper (or use the equivalent repo path while developing this skill):

     NOTEBOOKS_SKILL="${NOTEBOOKS_SKILL:-$HOME/.agents/skills/notebooks}"
     uv run --script "$NOTEBOOKS_SKILL/scripts/execute_notebook.py" \
       <notebook.ipynb> --kernel <project>
     

     The helper writes <notebook>.executed.ipynb. pixi run jupyter nbconvert --to notebook --execute --inplace <notebook.ipynb> is also valid when Jupyter is declared in the project environment.

7. Evaluate the plots, then refine. This step is required, not optional. After the run-all execution:

   • Open the executed notebook (or exported HTML) and visually inspect every figure.
   • Check for: empty axes, mis-scaled axes (log when linear was intended or vice versa), missing labels/legends, overlapping ticks, illegible font sizes at target output size, ambiguous palettes, colorbars without units, NaN-driven gaps, axis ranges clipping data, broken layouts.
   • For manuscript/paper figures, remove all in-plot titles and subtitles. Use axis labels, legends, panel letters, and manuscript captions instead.
   • Place each figure's caption/legend BELOW the figure: the figure (code) cell comes first and the caption (markdown) cell immediately follows it — never put the caption above the figure. A reader sees the figure, then its legend (journal convention).
   • If a figure is wrong or unclear, edit the source cell and re-run end-to-end. Repeat until each figure communicates what the surrounding markdown says it communicates.
   • Record what changed between revisions in a brief "Figure revision log" markdown cell or in the run log.

8. Deliver pre-executed notebooks. The artifact handed back to the user must:

   • Have every cell executed against the registered kernel.
   • Embed every figure (PNG / SVG cell outputs) directly in the .ipynb (or in the marimo HTML export).
   • Be reproducible from a clean clone: a new environment built from the PEP 723 header (marimo) or pixi install + jupyter nbconvert --to notebook --execute (Jupyter) must reproduce the same notebook end-to-end.

9. Convert between marimo and Jupyter when the user asks for it:

   • Use scripts/convert_notebook.py for a pinned conversion path. The fixtures under fixtures/ exercise both directions. Replace <PROJECT_NAME> and <PIXI_PROJECT_KERNEL> in copied Jupyter templates with the actual Pixi kernel before execution.
   • .ipynb → marimo .py: uvx marimo convert <notebook.ipynb> -o <notebook.py>, then uvx marimo check --strict <notebook.py>, then clean up Jupyter artifacts (display() calls, %magics, indented final expressions, ipywidget usage). See references/widgets.md and references/latex.md for ipywidget→marimo and MathJax→KaTeX mappings.
   • marimo .py → .ipynb: uvx marimo export ipynb <notebook.py> -o <notebook.ipynb> --include-outputs --sandbox -f.
   • After conversion, re-run step 6 (check/execute), step 7 (inspect plots), and step 8 (deliver pre-executed).

Quick Reference

Input Requirements

• Notebook scope and goals (what question, what data, what output).
• Data file paths (TSV/Parquet preferred for DuckDB ingestion).
• For marimo: uv available on PATH, or marimo installed in the environment.
• For Jupyter: pixi available and a pixi.toml (or equivalent env spec) for the project.

Output

• A reproducible notebook (.py for marimo, .ipynb for Jupyter) with narrative markdown cells, code cells, and embedded figures.
• A pre-executed copy (<notebook>.executed.ipynb or an .html export) where every cell has been run on a fresh kernel.
• A short "Figure revision log" recording any plot-revision rounds.

Quality Gates

• Notebook format chosen explicitly (marimo by default; Jupyter only when justified or when the input is .ipynb).
• Kernel registered and pinned: marimo PEP 723 header complete, or Jupyter kernelspec set to a named pixi kernel.
• Every Python import used in the notebook is declared in the dependency spec (PEP 723 header or pixi.toml).
• Data paths are project-relative and verified to exist.
• Headless run-all succeeds on a fresh kernel: marimo export uses --include-outputs --sandbox, or Jupyter execution exits zero.
• Every figure is inspected after execution; any figure that fails the visual checks above triggers a code revision and re-run.
• Manuscript/paper figures have no in-plot titles or subtitles.
• Delivered notebook has every cell pre-executed with figures embedded; users do not have to run the notebook to see the plots.
• For marimo: uvx marimo check --strict <notebook.py> passes before and after export.
• For marimo: no malformed markdown cells, quoted-string fragments inside mo.md(...), trailing empty cells, or return-only placeholder cells remain.
• Template dependencies use bounded versions and no delivered notebook retains the literal <PIXI_PROJECT_KERNEL> placeholder.
• Jupyter-to-marimo and marimo-to-Jupyter conversions both pass the fixture gate.

Examples

Example 1: New marimo notebook

# /// script
# requires-python = ">=3.12"
# dependencies = ["marimo", "polars", "duckdb", "matplotlib"]
# ///

import marimo
app = marimo.App(width="medium")

@app.cell
def _():
    import marimo as mo
    import polars as pl
    import duckdb
    import matplotlib.pyplot as plt
    return mo, pl, duckdb, plt

@app.cell(hide_code=True)
def _(mo):
    mo.md(r"""
    # Analysis notebook

    This notebook loads project data, validates it, and renders the requested figures.
    """)
    return

@app.cell
def _(duckdb):
    df = duckdb.read_parquet("data/measurements.parquet").pl()
    df.head()
    return (df,)

@app.cell
def _(df, plt):
    fig, ax = plt.subplots(figsize=(5, 3.2))
    ax.scatter(df["x"], df["y"], s=10)
    ax.set_xlabel("x (units)"); ax.set_ylabel("y (units)")
    fig
    return

Then:

uvx marimo check --strict notebook.py
uvx marimo export ipynb notebook.py -o notebook.executed.ipynb \
  --include-outputs --sandbox -f
uvx marimo check --strict notebook.py

Example 2: Jupyter notebook with a named pixi kernel

# One-time kernel registration in the project root:
pixi run python -m ipykernel install --user --name myproject --display-name "myproject (pixi)"

# After authoring, run end-to-end on a fresh kernel:
NOTEBOOKS_SKILL="${NOTEBOOKS_SKILL:-$HOME/.agents/skills/notebooks}"
uv run --script "$NOTEBOOKS_SKILL/scripts/execute_notebook.py" notebooks/analysis.ipynb \
  --kernel myproject \
  --out notebooks/analysis.executed.ipynb

Example 3: Convert .ipynb to marimo

uvx marimo convert notebooks/legacy.ipynb -o notebooks/legacy.py
uvx marimo check --strict notebooks/legacy.py
uvx marimo export ipynb notebooks/legacy.py -o notebooks/legacy.executed.ipynb \
  --include-outputs --sandbox -f

Troubleshooting

Issue: Jupyter notebook executes locally but fails on a teammate's machine. Solution: The kernel was unpinned (python3) or used a packaged interpreter outside the project's pixi env. Re-register a named kernel and pin it in the notebook kernelspec.

Issue: Marimo cell does not render a figure. Solution: The figure must be the final expression of the cell. Indented expressions inside if blocks or expressions buried before other statements will not render.

Issue: Figures look correct interactively but the executed file shows empty plots. Solution: Code is mutating shared state across cells (e.g. plt.gcf() reuse). Build a fresh fig, ax = plt.subplots(...) per cell and return / display fig as the final expression.

Issue: Converted notebook fails marimo check. Solution: Remove leftover display(...) calls, drop %magic lines that have no marimo equivalent, and rewrite ipywidget usage using mo.ui.* per references/widgets.md.

Issue: Every matplotlib figure appears twice in the executed Jupyter notebook. Solution: The inline backend's flush_figures post-execute hook auto-displays every open figure (display_data), and the cell's fig return value produces a second copy (execute_result). Unregister the hook in the preamble cell:

plt.ioff()
try:
    from matplotlib_inline.backend_inline import flush_figures
    get_ipython().events.unregister("post_execute", flush_figures)
except Exception:
    pass

With this fix, only the cell's final fig expression renders. For figures inside if/else blocks (where fig is not a top-level expression and therefore not captured as execute_result), use display(fig) explicitly instead of bare fig. Do not use mpl.use("agg") as a workaround — it disables _repr_png_() entirely and produces zero images.