Add Pipeline Story

Add Pipeline Story

Add a new pipeline example to the mthds-ui Storybook from a .mthds bundle file. Always generates BOTH dry-run and live-run GraphSpecs.

Prerequisites

• pipelex CLI must be on PATH
• mthds-agent CLI must be on PATH (for input generation)
• The .mthds file must be accessible from the workspace

Workflow

1. Identify the .mthds file

Ask the user for the path to the .mthds bundle if not provided. Verify the file exists.

2. Determine the next pipeline number

ls data/pipelines/ | grep pipeline | sort -t_ -k2 -n | tail -1

Extract the number (e.g., pipeline_30 -> next is pipeline_31). The story file number matches: Pipeline31.stories.tsx.

3. Choose a spec name

Derive a spec name from the bundle filename or ask the user.

• Bundle cv_analyzer.mthds -> spec name cvAnalyzer, const prefix CV_ANALYZER
• Bundle report_writer/bundle.mthds -> spec name reportWriter, const prefix REPORT_WRITER
• Use camelCase for filename, UPPER_SNAKE_CASE for const prefix

4. Create the data directory

mkdir -p data/pipelines/pipeline_<NN>
cp <path-to-bundle.mthds> data/pipelines/pipeline_<NN>/bundle.mthds

5. Generate inputs

Use mthds-agent to discover what inputs the bundle requires:

mthds-agent inputs bundle data/pipelines/pipeline_<NN>/bundle.mthds -L data/pipelines/pipeline_<NN>/

This outputs a JSON structure showing required inputs with their concepts. The generated URLs are mock URLs (https://mock-*.invalid/...).

Replace mock URLs with real, accessible test files. Use files from:

• https://pipelex-web.s3.amazonaws.com/demo/ for PDFs (John-Doe-CV.pdf, Job-Offer.pdf)
• https://pipelex-web.s3.us-west-2.amazonaws.com/tests/ for images (alan_turing.jpg, eiffel_tower.jpg)
• Or ask the user for specific input files

Save the inputs:

# Write the inputs JSON to the data directory
cat > data/pipelines/pipeline_<NN>/inputs.json << 'EOF'
{
  ...inputs JSON with real URLs...
}
EOF

6. Generate GraphSpecs (ALWAYS both dry-run AND live-run)

Dry run:

pipelex run bundle data/pipelines/pipeline_<NN>/bundle.mthds \
  --dry-run --mock-inputs --graph \
  -o /tmp/pipeline<NN>_dry

Copy the output:

cp /tmp/pipeline<NN>_dry/<output_dir>/graphspec.json \
  data/pipelines/pipeline_<NN>/dry_run_graph_spec.json

Live run:

pipelex run bundle data/pipelines/pipeline_<NN>/bundle.mthds \
  --graph \
  -i data/pipelines/pipeline_<NN>/inputs.json \
  -o /tmp/pipeline<NN>_live

Copy the output:

cp /tmp/pipeline<NN>_live/<output_dir>/graphspec.json \
  data/pipelines/pipeline_<NN>/live_run_graph_spec.json

7. Create the spec file

Create src/graph/react/viewer/__stories__/pipelines/specs/<specName>.ts:

Use Python to convert JSON to TypeScript (handles null/undefined properly):

python3 -c "
import json

with open('data/pipelines/pipeline_<NN>/dry_run_graph_spec.json') as f:
    dry = json.load(f)
with open('data/pipelines/pipeline_<NN>/live_run_graph_spec.json') as f:
    live = json.load(f)

content = 'import type { GraphSpec } from \"@graph/types\";\n\n'
content += '// Generated by: pipelex run bundle data/pipelines/pipeline_<NN>/bundle.mthds\n'
content += '// Do not edit manually — regenerate by running the pipeline.\n\n'
content += 'export const DRY_<CONST_PREFIX>: GraphSpec = ' + json.dumps(dry, indent=2, ensure_ascii=False) + ' as any as GraphSpec;\n\n'
content += 'export const LIVE_<CONST_PREFIX>: GraphSpec = ' + json.dumps(live, indent=2, ensure_ascii=False) + ' as any as GraphSpec;\n'

with open('src/graph/react/viewer/__stories__/pipelines/specs/<specName>.ts', 'w') as f:
    f.write(content)
"

The as any as GraphSpec cast handles null values from pipelex output that TypeScript types define as undefined.

Run prettier on the file:

npx prettier --write src/graph/react/viewer/__stories__/pipelines/specs/<specName>.ts

8. Wire into mockGraphSpec.ts

Edit src/graph/react/viewer/__stories__/mockGraphSpec.ts:

1. Add import after the last import block:

   import { DRY_<CONST_PREFIX>, LIVE_<CONST_PREFIX> } from "./pipelines/specs/<specName>";
   

2. Add to the re-export block (inside export { ... }):

   DRY_<CONST_PREFIX>,
   LIVE_<CONST_PREFIX>,
   

9. Create the story file

Create src/graph/react/viewer/__stories__/pipelines/Pipeline<NN>.stories.tsx:

import type { Meta, StoryObj } from "@storybook/react-vite";
import { GraphViewer } from "../../GraphViewer";
import { DRY_<CONST_PREFIX>, LIVE_<CONST_PREFIX> } from "../mockGraphSpec";

const meta: Meta<typeof GraphViewer> = {
  title: "Graph/GraphViewer/<NN> <Human-Readable Title>",
  component: GraphViewer,
  decorators: [
    (Story) => (
      <div style={{ width: "100%", height: "100vh", position: "relative" }}>
        <Story />
      </div>
    ),
  ],
  argTypes: {
    direction: { control: { type: "inline-radio" }, options: ["LR", "TB"] },
    showControllers: { control: { type: "boolean" } },
  },
};

export default meta;
type Story = StoryObj<typeof GraphViewer>;

const D = { direction: "LR" as const, showControllers: true };

export const DryRun: Story = {
  args: { graphspec: DRY_<CONST_PREFIX>, ...D },
};

export const LiveRun: Story = {
  args: { graphspec: LIVE_<CONST_PREFIX>, ...D },
};

10. Validate

make check

Fix any lint/format/type errors before finishing.

11. Report

Tell the user:

• The data directory: data/pipelines/pipeline_<NN>/
• The Storybook story: Graph/GraphViewer/<NN> <Title>
• How to view: make storybook then navigate to the story
• Remind: never hand-edit spec files, regenerate with pipelex run bundle

Data directory structure

Each pipeline example lives in data/pipelines/pipeline_<NN>/:

data/pipelines/pipeline_<NN>/
  bundle.mthds                # The source .mthds bundle          — hand-authored
  inputs.json                 # Inputs for the live run           — hand-authored
  inputs/                     # Input files, when the pipe takes any — hand-authored
  dry_run_graph_spec.json     # GraphSpec from the dry run        — generated
  dry_run_graph.html          # Standalone ReactFlow viewer       — generated
  dry_run_mermaidflow.mmd     # Mermaid source                    — generated
  dry_run_mermaidflow.html    # Rendered Mermaid page             — generated
  live_run_graph_spec.json    # GraphSpec from the live run       — generated
  live_run_graph.html         # Standalone ReactFlow viewer       — generated
  live_run_mermaidflow.mmd    # Mermaid source                    — generated
  live_run_mermaidflow.html   # Rendered Mermaid page             — generated
  live_run_main_stuff.json    # What the live run produced        — generated
  inputs_template.json        # Fill-in template for inputs.json  — generated

Only bundle.mthds, inputs.json and inputs/ are authored. Everything else is written by scripts/generate-fixtures.mjs — never hand-edit a generated file, and never add a generated file the script does not write. Regenerate with make fixtures> (free) and make fixtures-live> (real inference).