Pipeline diagram
Render a clean pipeline diagram in a flowing-backbone style (inputs branch in from
the left, steps run down a tan backbone with the key decision of each step as a blue
annotation, outputs branch out to the right with their file names). Spec-driven: you
describe the pipeline in a short YAML file and a generic renderer draws it.
When to use
- The user wants to see / communicate how a pipeline works — its steps, the key
decisions made at each step, and which files go in and out.
- After building a multi-script analysis, to produce an overview figure (and per-step
detail) for a talk, methods section, lab onboarding, or your own understanding.
Do not use for the result figures (networks/heatmaps/trees), or for plain
flowcharts with no data-pipeline structure.
Workflow
- Copy the spec template into the project (e.g. into
scripts/<area>/ or outs/):
templates/pipeline_spec.example.yaml → pipeline.yaml.
- Fill in the steps (top → bottom = execution order). Per step, set
title and the
key decision (decision); add inputs/outputs for the overview and
script/files_in/files_out/params for the detail cards. Build the spec by
reading the project's numbered scripts: their input block (top of file), their
outs/XX_*/ outputs, and the genuine decisions they encode (thresholds, joins,
filters, what's assumed). Surface the real decisions — that's the point of the
blue annotations, not a restating of the title.
- Render (needs
matplotlib + pyyaml — use the project conda env):python templates/render_pipeline.py pipeline.yaml outs/<area>/pipeline_overview --detail
Writes <prefix>.png/.pdf (overview) and, with --detail, <prefix>_detail.png/.pdf.
- Look at the PNG and iterate. Layout is auto-computed (steps evenly spaced;
inputs/outputs greedily spread to avoid label collisions; connectors fan by
distance). If two labels still touch, nudge a step's input order or split a long step.
Spec schema (per step)
| field |
used by |
meaning |
title |
both |
step name (bold) — required |
decision |
overview |
the KEY decision/assumption (blue italic). Use \n to wrap. |
inputs |
overview |
list of {label, kind: required|optional} (left nodes) |
outputs |
overview |
list of {label, file, terminal} (right nodes; file shown in monospace; terminal: true = purple end node) |
script |
detail |
script filename (card header) |
files_in / files_out |
detail |
exact file paths/names (basenames read best) |
params |
detail |
key params / decisions as short bullets |
Top-level: title, subtitle, optional theme: (override any palette colour, e.g.
theme: {line: '#888', blue: '#2a6'}).
Tips / gotchas
- Spec-driven, not inferred. Writing the spec forces you to state the real I/O and
decisions; auto-parsing scripts is brittle. Draft from the scripts, then refine.
- Keep
decision text to the genuine choice (threshold, what's excluded, what's
assumed) — e.g. "trust an edge only if recovered in human", not "score the edges".
- Long file paths in
files_in read best as basenames; the detail card puts files
and decisions on separate rows, but very long names still crowd — shorten them.
- Node types in the legend appear only if present (required/optional inputs, terminal).
- A worked example is
templates/pipeline_spec.example.yaml (the synapse-PPI pipeline).
Files
templates/render_pipeline.py — generic renderer (overview + optional detail).templates/pipeline_spec.example.yaml — annotated example spec + schema.
1---2name: pipeline-diagram3description: Generate a publication-style processing-pipeline diagram from a small YAML spec: a flowing-backbone OVERVIEW (steps down a backbone, inputs branching in from the left, key decisions as blue annotations, output files branching right) plus an optional PER-STEP DETAIL view (one card per step showing input files -> key params/decisions -> output files). Use when the user wants to diagram, visualize, map, or document an analysis/processing pipeline or multi-script workflow; create a pipeline figure / schematic / flow diagram; or show/communicate the steps, key decisions, and input/output files of a pipeline. Spec-driven by design (you write a short YAML, not auto-parsed from code). Do NOT load for the analysis RESULT figures themselves (networks, heatmaps, trees), for generic flowcharts unrelated to a data pipeline, or for Quarto/script scaffolding (use script-organization).4---56# Pipeline diagram78Render a clean pipeline diagram in a flowing-backbone style (inputs branch in from9the left, steps run down a tan backbone with the key decision of each step as a blue10annotation, outputs branch out to the right with their file names). Spec-driven: you11describe the pipeline in a short YAML file and a generic renderer draws it.1213## When to use1415- The user wants to **see / communicate how a pipeline works** — its steps, the key16 decisions made at each step, and which files go in and out.17- After building a multi-script analysis, to produce an overview figure (and per-step18 detail) for a talk, methods section, lab onboarding, or your own understanding.1920Do **not** use for the result figures (networks/heatmaps/trees), or for plain21flowcharts with no data-pipeline structure.2223## Workflow24251. **Copy the spec template** into the project (e.g. into `scripts/<area>/` or `outs/`):26 `templates/pipeline_spec.example.yaml` → `pipeline.yaml`.272. **Fill in the steps** (top → bottom = execution order). Per step, set `title` and the28 key decision (`decision`); add `inputs`/`outputs` for the overview and29 `script`/`files_in`/`files_out`/`params` for the detail cards. Build the spec by30 reading the project's numbered scripts: their input block (top of file), their31 `outs/XX_*/` outputs, and the genuine decisions they encode (thresholds, joins,32 filters, what's assumed). **Surface the real decisions** — that's the point of the33 blue annotations, not a restating of the title.343. **Render** (needs `matplotlib` + `pyyaml` — use the project conda env):35 ```bash36 python templates/render_pipeline.py pipeline.yaml outs/<area>/pipeline_overview --detail37 ```38 Writes `<prefix>.png/.pdf` (overview) and, with `--detail`, `<prefix>_detail.png/.pdf`.394. **Look at the PNG and iterate.** Layout is auto-computed (steps evenly spaced;40 inputs/outputs greedily spread to avoid label collisions; connectors fan by41 distance). If two labels still touch, nudge a step's input order or split a long step.4243## Spec schema (per step)4445| field | used by | meaning |46|---|---|---|47| `title` | both | step name (bold) — required |48| `decision` | overview | the KEY decision/assumption (blue italic). Use `\n` to wrap. |49| `inputs` | overview | list of `{label, kind: required\|optional}` (left nodes) |50| `outputs` | overview | list of `{label, file, terminal}` (right nodes; `file` shown in monospace; `terminal: true` = purple end node) |51| `script` | detail | script filename (card header) |52| `files_in` / `files_out` | detail | exact file paths/names (basenames read best) |53| `params` | detail | key params / decisions as short bullets |5455Top-level: `title`, `subtitle`, optional `theme:` (override any palette colour, e.g.56`theme: {line: '#888', blue: '#2a6'}`).5758## Tips / gotchas5960- **Spec-driven, not inferred.** Writing the spec forces you to state the real I/O and61 decisions; auto-parsing scripts is brittle. Draft from the scripts, then refine.62- **Keep `decision` text to the genuine choice** (threshold, what's excluded, what's63 assumed) — e.g. "trust an edge only if recovered in human", not "score the edges".64- **Long file paths** in `files_in` read best as basenames; the detail card puts files65 and decisions on separate rows, but very long names still crowd — shorten them.66- **Node types** in the legend appear only if present (required/optional inputs, terminal).67- A worked example is `templates/pipeline_spec.example.yaml` (the synapse-PPI pipeline).6869## Files7071- `templates/render_pipeline.py` — generic renderer (`overview` + optional `detail`).72- `templates/pipeline_spec.example.yaml` — annotated example spec + schema.
MusserLab