Pipeline State — Persistence & Recovery
State Location
Schema Definition
Atomic Write Protocol
Encoding invariant (Q12)
State file MUST be UTF-8 without BOM. The byte at offset 0 MUST be 0x7B (the { character that opens a JSON object), NOT 0xEF (the first byte of the UTF-8 BOM sequence EF BB BF). Every implementation below guarantees this. A state file beginning with a BOM is unparseable by JSON.parse in Claude Code and produces a Parse Error → Corruption detected. Escalate to human recovery action. The protocol is shell-specific because naïve ports introduce BOMs silently.
Per-shell implementations
Bash / zsh:
TEMP_DIR="${SCOPE_ROOT}/superpipelines/temp/${PIPELINE_NAME}/${RUN_ID}"
mkdir -p "$TEMP_DIR"
# printf (not echo) — echo's newline behavior varies by shell and locale.
printf '%s' "$NEW_STATE_JSON" > "${TEMP_DIR}/pipeline-state.json.tmp"
mv "${TEMP_DIR}/pipeline-state.json.tmp" "${TEMP_DIR}/pipeline-state.json"
PowerShell (Windows):
$TempDir = "$ScopeRoot/superpipelines/temp/$PipelineName/$RunId"
New-Item -ItemType Directory -Force -Path $TempDir | Out-Null
# The $false constructor argument disables BOM. Never use `Set-Content -Encoding UTF8` —
# in Windows PowerShell 5.1 that variant emits a BOM that breaks JSON.parse downstream.
[System.IO.File]::WriteAllText("$TempDir/pipeline-state.json.tmp", $NewStateJson, [System.Text.UTF8Encoding]::new($false))
Move-Item -Force "$TempDir/pipeline-state.json.tmp" "$TempDir/pipeline-state.json"
Node.js:
const fs = require('node:fs');
const path = require('node:path');
const tempDir = path.join(scopeRoot, 'superpipelines', 'temp', pipelineName, runId);
fs.mkdirSync(tempDir, { recursive: true });
// Node's 'utf8' encoding has no BOM by default.
fs.writeFileSync(path.join(tempDir, 'pipeline-state.json.tmp'), newStateJson, { encoding: 'utf8' });
fs.renameSync(path.join(tempDir, 'pipeline-state.json.tmp'), path.join(tempDir, 'pipeline-state.json'));
Byte-0 verification
A correct implementation can be verified by checking the first byte of the written file:
# Expect: 0x7B (the '{' character). Any other byte (especially 0xEF) is a defect.
od -An -tx1 -N1 pipeline-state.json
Recovery & Resumption Rules
This survives workspace moves between machines, between WSL and native Windows, and between drives. If the basename sanity check fails for the pipeline's layout, the state file has been moved out of a recognized scope and resume MUST surface the inconsistency.
Reference Files
sk-pipeline-paths/SKILL.md — Scope root resolution.sk-pipeline-patterns/SKILL.md — Execution pattern definitions.running-a-pipeline/SKILL.md — Primary orchestrator workflow.
1---2name: sk-pipeline-state3description: Defines the schema, storage layout, and recovery protocols for `pipeline-state.json`. Use when reading or writing pipeline state, resuming an interrupted run, or diagnosing a crashed orchestrator.4---56# Pipeline State — Persistence & Recovery78<overview>9Superpipelines use a structured JSON file to manage the lifecycle of multi-agent workflows. State is isolated from model behavior, ensuring runs are inspectable, resumable, and resilient to environment restarts. All state transitions follow an atomic write pattern to prevent corruption.10</overview>1112<glossary>13 <term name="Pipeline State">A structured JSON file (`pipeline-state.json`) representing the source of truth for a specific run.</term>14 <term name="Atomic Write">The process of writing to a temporary file and renaming it to ensure file integrity.</term>15 <term name="Run ID">A UUID v4 uniquely identifying a single execution instance of a pipeline.</term>16 <term name="source_tier">The execution tier where the pipeline was scaffolded. Set once at run init; never updated.</term>17 <term name="runtime_tier">The execution tier of the current or most-recent run. Re-detected on every resume; updated on cross-tier resume.</term>18 <term name="tier_changes">Append-only audit log of every cross-tier resume event. Never overwritten.</term>19</glossary>2021## State Location2223<invariant>24State must be persisted to `<scope-root>/superpipelines/temp/{P}/{runId}/pipeline-state.json`. Never store state within `${CLAUDE_PLUGIN_ROOT}`, as it is not persistent across updates.25</invariant>2627## Schema Definition2829<schema>30```json31{32 "pipeline_id": "<uuid>",33 "pipeline_name": "<P>",34 "plugin_version": "<semver — copied at run start from platform_profile.extensions.version_manifest_path (per-tier; see Q12)>",35 "scope_root_dir": "<directory NAME, NOT an absolute path (Q12 portability fix). For layout:data pipelines this is the CONSTANT '.superpipelines' (tier-independent, #64 collapse); for layout:legacy (pre-v2 old-root) pipelines it is the dir NAME from platform_profile.scope_root.workspace, e.g. '.claude'>",36 "run_id": "<uuid>",37 "started_at": "<iso8601>",38 "pattern": "1 | 2 | 2b | 3 | 4 | 5",39 "status": "running | completed | escalated | failed",40 "current_phase": <index>,41 "phases": [42 {43 "index": 0,44 "name": "<phase name>",45 "status": "pending | running | done | failed",46 "agent": "<agent name>",47 "outputs": ["<path>"],48 "error": null49 }50 ],51 "metadata": {52 "source_tier": "<tier_id — tier where pipeline was scaffolded; immutable after init>",53 "runtime_tier": "<tier_id — tier where current execution runs; re-detected on every resume>",54 "platform_profile": "<full profile object snapshot — updated when runtime_tier changes>",55 "tier_changes": [56 { "from": "<tier_id>", "to": "<tier_id>", "at": "<iso8601>" }57 ],58 "source_scope_root": "<original workspace scope root directory name, e.g. .claude>",59 "isolation_warning": "<joined degradation_warnings from active profile; null if none>",60 "resolved_models": {61 "<step_id>": "<resolved object from sk-model-resolver — model, effort, source, warnings>"62 },63 "preference_files_consulted": {64 "user_path": "<absolute or ~-prefixed path to user-global prefs>",65 "user_hash": "<sha256:hex digest of user prefs file content at Phase 0.45; null if file absent>",66 "workspace_path": "<absolute path to workspace prefs>",67 "workspace_hash": "<sha256:hex digest of workspace prefs file content at Phase 0.45; null if file absent>"68 },69 "model_tiers_version_at_run": "<profile.model_tiers_version at Phase 0.45; ISO date>"70 }71}72```73</schema>7475## Atomic Write Protocol7677<protocol>78To prevent JSON corruption during concurrent operations or crashes, always use the following atomic write pattern:791. Write the new state content to `pipeline-state.json.tmp`.802. Move (rename) the temporary file to `pipeline-state.json`.8182### Encoding invariant (Q12)8384State file MUST be **UTF-8 without BOM**. The byte at offset 0 MUST be `0x7B` (the `{` character that opens a JSON object), NOT `0xEF` (the first byte of the UTF-8 BOM sequence `EF BB BF`). Every implementation below guarantees this. A state file beginning with a BOM is unparseable by `JSON.parse` in Claude Code and produces a `Parse Error` → `Corruption detected. Escalate to human` recovery action. The protocol is shell-specific because naïve ports introduce BOMs silently.8586### Per-shell implementations8788**Bash / zsh:**89```bash90TEMP_DIR="${SCOPE_ROOT}/superpipelines/temp/${PIPELINE_NAME}/${RUN_ID}"91mkdir -p "$TEMP_DIR"92# printf (not echo) — echo's newline behavior varies by shell and locale.93printf '%s' "$NEW_STATE_JSON" > "${TEMP_DIR}/pipeline-state.json.tmp"94mv "${TEMP_DIR}/pipeline-state.json.tmp" "${TEMP_DIR}/pipeline-state.json"95```9697**PowerShell (Windows):**98```powershell99$TempDir = "$ScopeRoot/superpipelines/temp/$PipelineName/$RunId"100New-Item -ItemType Directory -Force -Path $TempDir | Out-Null101# The $false constructor argument disables BOM. Never use `Set-Content -Encoding UTF8` —102# in Windows PowerShell 5.1 that variant emits a BOM that breaks JSON.parse downstream.103[System.IO.File]::WriteAllText("$TempDir/pipeline-state.json.tmp", $NewStateJson, [System.Text.UTF8Encoding]::new($false))104Move-Item -Force "$TempDir/pipeline-state.json.tmp" "$TempDir/pipeline-state.json"105```106107**Node.js:**108```js109const fs = require('node:fs');110const path = require('node:path');111const tempDir = path.join(scopeRoot, 'superpipelines', 'temp', pipelineName, runId);112fs.mkdirSync(tempDir, { recursive: true });113// Node's 'utf8' encoding has no BOM by default.114fs.writeFileSync(path.join(tempDir, 'pipeline-state.json.tmp'), newStateJson, { encoding: 'utf8' });115fs.renameSync(path.join(tempDir, 'pipeline-state.json.tmp'), path.join(tempDir, 'pipeline-state.json'));116```117118### Byte-0 verification119120A correct implementation can be verified by checking the first byte of the written file:121122```bash123# Expect: 0x7B (the '{' character). Any other byte (especially 0xEF) is a defect.124od -An -tx1 -N1 pipeline-state.json125```126</protocol>127128## Recovery & Resumption Rules129130<recovery_rules>131| State Found | Required Action |132| :--- | :--- |133| **`status: running`** (<1h old) | Active run detected; refuse to start a new instance. |134| **`status: running`** (>1h old) | Treat as crashed. Prompt user to resume, restart, or abort. |135| **`status: completed`** | Terminal state reached. Skip or archive. |136| **`status: escalated/failed`** | Stop execution. Surface to human for manual intervention. |137| **Parse Error** | Corruption detected. Escalate to human; do NOT auto-resume. |138</recovery_rules>139140<invariants>141- **No Model Coupling**: Never use the model's native memory tool for pipeline state management; use structured JSON.142- **Atomic Renaming**: Direct writes to `pipeline-state.json` are forbidden.143- **Explicit Resumption**: NEVER auto-resume from an `escalated` or `failed` state without explicit user confirmation.144- **Backward Compatibility**: Pre-v2.0.0 state files carry `metadata.tier` (single field). On resume of an old state file: treat `metadata.tier` as `source_tier` when `metadata.source_tier` is absent; set `runtime_tier` to the re-detected current tier. New state writes MUST use `source_tier` and `runtime_tier`; never write `metadata.tier` in new state.145- **Version Stamping (Q12)**: `plugin_version` MUST be set at state initialization by reading the `version` field from the per-tier manifest at `platform_profile.extensions.version_manifest_path`. The Tier 1 manifest is `.claude-plugin/plugin.json`; other tiers point to their own manifest (Codex `.codex-plugin/plugin.json`, Cursor `.cursor-plugin/plugin.json`, OpenCode `opencode-plugin.json`, Antigravity `gemini-extension.json` until retirement). It is read-only after init and used by `running-a-pipeline` for compatibility advisory.146- **Portable scope_root (Q12 + #64 collapse)**: State files store `scope_root_dir` (the directory NAME) instead of the previous absolute path. On resume, the active scope_root absolute path is recomputed from the state file's own location. The recompute depth is **layout-dependent** because data-only drops the `superpipelines/` path infix:147 - `layout:data` — state lives at `{scope_root}/temp/{P}/{runId}/pipeline-state.json` (`scope_root` = the `.superpipelines` DATA_ROOT). Recompute `scope_root = dirname^4(state_file_path)`; the sanity check `basename(scope_root) == scope_root_dir` yields `.superpipelines`.148 - `layout:legacy` — state lives at `{scope_root}/superpipelines/temp/{P}/{runId}/pipeline-state.json` (`scope_root` = e.g. `.claude`). The extra `superpipelines/` infix puts `scope_root` one level higher than the data case: recompute `scope_root = dirname^5(state_file_path)` (vs `^4` for data); the sanity check `basename(scope_root) == scope_root_dir` yields e.g. `.claude`.149 150 This survives workspace moves between machines, between WSL and native Windows, and between drives. If the basename sanity check fails for the pipeline's layout, the state file has been moved out of a recognized scope and resume MUST surface the inconsistency.151</invariants>152153## Reference Files154155- `sk-pipeline-paths/SKILL.md` — Scope root resolution.156- `sk-pipeline-patterns/SKILL.md` — Execution pattern definitions.157- `running-a-pipeline/SKILL.md` — Primary orchestrator workflow.
gustavo-meilus