ceci — DESC pipeline framework
You're helping a user write code with ceci, the DESC pipeline framework.
Most DESC analysis packages (TXPipe, RAIL, etc.) are built on ceci.
For NERSC job submission and cluster conventions, see the nersc skill.
Overview
A stage subclasses PipelineStage, declares inputs/outputs as lists of (tag, FileType) tuples, sets config_options, and implements run().
A pipeline is a YAML file: modules, launcher, site, stages, inputs, output_dir, log_dir, config.
Ceci resolves execution order from the tag DAG automatically — YAML order is irrelevant.
from ceci import PipelineStage
from ceci.file_types import HDFFile, TextFile
class MyStage(PipelineStage):
name = "MyStage"
inputs = [("input_cat", HDFFile)]
outputs = [("output_cat", HDFFile), ("summary", TextFile)]
config_options = {"quality_cut": float, "n_bins": 10}
def run(self):
cut = self.config["quality_cut"]
with self.open_input("input_cat", wrapper=True) as f:
data = f.file["shear/g1"][:]
with self.open_output("output_cat", wrapper=True) as out:
out.file.create_dataset("result/g1", data=data[data > cut])
Key rules: name is required and globally unique; all declared outputs must be written on every run; resume: true in YAML skips stages with existing outputs; nprocess: N triggers MPI with N ranks.
Reference
| Topic |
Reference |
| Stage definition, file types, MPI, pipeline YAML, CLI, gotchas |
references/stages-and-pipelines.md |
| NERSC launchers, parsl site config, interactive vs batch, per-stage resources, parsl gotchas |
references/nersc-parsl.md |
1---2name: ceci3description: How to write DESC pipeline stages and pipelines with ceci. Covers PipelineStage subclassing, inputs/outputs declarations, config_options, HDFFile/FitsFile/TextFile/YamlFile I/O, open_input/open_output, parallel iteration helpers, MPI patterns, pipeline YAML structure (modules/launcher/site/stages/inputs), stage aliasing, CLI usage, resume mode, nprocess, and NERSC/parsl deployment. Use this skill whenever the user imports ceci; subclasses PipelineStage; defines inputs, outputs, or config_options on a stage; calls open_input or open_output; writes or debugs a pipeline YAML; runs ceci from the CLI; asks about stage aliasing, DuplicateStageName, resume mode, nprocess/MPI, parsl launchers, or ceci file types.4---56# ceci — DESC pipeline framework78You're helping a user write code with ceci, the DESC pipeline framework.9Most DESC analysis packages (TXPipe, RAIL, etc.) are built on ceci.10For NERSC job submission and cluster conventions, see the `nersc` skill.1112## Overview1314A **stage** subclasses `PipelineStage`, declares `inputs`/`outputs` as lists of `(tag, FileType)` tuples, sets `config_options`, and implements `run()`.15A **pipeline** is a YAML file: `modules`, `launcher`, `site`, `stages`, `inputs`, `output_dir`, `log_dir`, `config`.16Ceci resolves execution order from the tag DAG automatically — YAML order is irrelevant.1718```python19from ceci import PipelineStage20from ceci.file_types import HDFFile, TextFile2122class MyStage(PipelineStage):23 name = "MyStage"24 inputs = [("input_cat", HDFFile)]25 outputs = [("output_cat", HDFFile), ("summary", TextFile)]26 config_options = {"quality_cut": float, "n_bins": 10}2728 def run(self):29 cut = self.config["quality_cut"]30 with self.open_input("input_cat", wrapper=True) as f:31 data = f.file["shear/g1"][:]32 with self.open_output("output_cat", wrapper=True) as out:33 out.file.create_dataset("result/g1", data=data[data > cut])34```3536Key rules: `name` is required and globally unique; all declared outputs must be written on every run; `resume: true` in YAML skips stages with existing outputs; `nprocess: N` triggers MPI with N ranks.3738## Reference3940| Topic | Reference |41|---|---|42| Stage definition, file types, MPI, pipeline YAML, CLI, gotchas | [`references/stages-and-pipelines.md`](references/stages-and-pipelines.md) |43| NERSC launchers, parsl site config, interactive vs batch, per-stage resources, parsl gotchas | [`references/nersc-parsl.md`](references/nersc-parsl.md) |
jfcrenshaw