Streaming Output

Streaming Output

Write long-form content incrementally to markdown files with automatic resume capability. If context limits are hit mid-generation, work persists and can be continued.

⚠️ MANDATORY USE CASES

ALWAYS use this skill when:

• Generating B-SPEC documents (typically 1,500-4,000 lines)
• Writing any document expected to exceed 1,000 lines
• Creating multi-section specifications or reports
• Context compaction has already occurred in the conversation
• The continuation prompt mentions streaming-output

DO NOT use manual heredoc appends (cat >> file << 'EOF') for long documents. This pattern fails silently when context compaction occurs mid-write, causing content corruption that is difficult to detect and repair.

Commands

Quick Start

# 1. Initialize with a plan
/stream.init report.md --sections "intro,methodology,findings,conclusion"

# 2. Write sections (repeat for each)
/stream.write intro
/stream.write methodology
# ... if interrupted, resume later with:
/stream.resume

# 3. Finalize when complete
/stream.finalize

/stream.init

Initialize an output file with a section plan.

Usage: /stream.init <filepath> --sections "<comma-separated-list>" [--template <template-name>]

Workflow:

1. Create output file (or detect existing)
2. Write header with section plan as YAML frontmatter
3. Generate content hash placeholder for integrity checking
4. Present checklist for tracking

Templates (optional):

• bspec - 15-section B-SPEC structure
• report - Standard report structure
• spec - Generic specification structure

Example:

# Standard initialization
python scripts/stream_write.py init report.md \
  --sections "introduction,background,analysis,recommendations,conclusion"

# B-SPEC template (pre-defined 15 sections)
python scripts/stream_write.py init b-spec-010.md --template bspec

Output file structure:

---
stream_plan:
  version: "2.0"
  sections:
    - id: introduction
      status: pending
      hash: null
    - id: background
      status: pending
      hash: null
    - id: analysis
      status: pending
      hash: null
  created: 2024-01-15T10:30:00
  last_modified: null
  integrity_check: true
---

# Report

<!-- Content will be streamed below -->

/stream.write

Write a single section to the file with markers and integrity verification.

Usage: /stream.write <section-id>

Workflow:

1. Check section exists in plan and is pending
2. Generate content for section
3. Compute content hash
4. Write to temporary location first
5. Validate write completed (check for SECTION_END marker)
6. Append to main file with SECTION_START and SECTION_END markers
7. Update section status to completed with hash

Script:

python scripts/stream_write.py write report.md introduction "Your content here..."

Markers in file:

<!-- SECTION_START: introduction | hash:a1b2c3d4 -->
## Introduction

Your introduction content here...

<!-- SECTION_END: introduction | hash:a1b2c3d4 -->

Important:

• Write ONE section at a time
• Verify success before proceeding
• Hash in START and END markers must match (integrity check)

Write Verification

After each write, the skill automatically verifies:

1. SECTION_START marker exists
2. SECTION_END marker exists
3. Hashes in both markers match
4. Content between markers is non-empty

If verification fails, the write is flagged and /stream.repair is recommended.

/stream.status

Show current progress, identify resume point, and check integrity.

Usage: /stream.status <filepath> [--verify]

Script:

python scripts/stream_status.py report.md
python scripts/stream_status.py report.md --verify  # Full integrity check

Standard Output:

Stream Status: report.md

Sections:
  [x] introduction (completed) ✓
  [x] background (completed) ✓
  [ ] analysis (pending) <- RESUME HERE
  [ ] recommendations (pending)
  [ ] conclusion (pending)

Progress: 2/5 sections (40%)
Next section: analysis

With --verify flag (integrity check):

Stream Status: report.md

Integrity Check:
  [x] introduction - hash:a1b2c3d4 ✓ valid
  [x] background - hash:e5f6g7h8 ✓ valid
  [!] analysis - CORRUPTED (START without END)
  [ ] recommendations - pending
  [ ] conclusion - pending

⚠️  CORRUPTION DETECTED in section: analysis
    Run `/stream.repair analysis` to fix

Progress: 2/5 sections (40%)
Next section: analysis (requires repair)

Corruption Detection

The status command detects:

• Orphaned START: SECTION_START exists without matching SECTION_END
• Hash mismatch: START and END marker hashes don't match
• Empty section: Markers exist but no content between them
• Duplicate sections: Same section ID appears multiple times

/stream.resume

Continue writing from the last incomplete section.

Usage: /stream.resume <filepath>

Workflow:

1. Run status with verification to find resume point
2. Check for corrupted sections (repair if needed)
3. Read existing content for context
4. Continue with /stream.write for next pending section

Script:

python scripts/stream_status.py report.md --resume

Output:

Resume Point: report.md

Last completed: background
Next pending: analysis

Context from previous sections loaded (2,450 tokens)
Ready to write: analysis

Command: /stream.write analysis

If corruption is detected:

Resume Point: report.md

⚠️  CORRUPTION DETECTED
Section 'analysis' has incomplete markers.

Recommended action:
1. Run `/stream.repair analysis` to remove partial content
2. Then run `/stream.write analysis` to regenerate

Command: /stream.repair analysis

/stream.repair

Fix corrupted or partial sections.

Usage: /stream.repair <filepath> <section-id> [--strategy <strategy>]

Strategies:

• remove (default): Remove partial content, reset section to pending
• complete: Attempt to add missing END marker (use with caution)
• backup: Create backup before repair

Workflow:

1. Create backup of current file (if --strategy backup)
2. Locate corrupted section
3. Remove content from SECTION_START to end of partial content
4. Update section status to pending
5. Report repair results

Script:

python scripts/stream_repair.py report.md analysis --strategy remove

Output:

Repair Report: report.md

Section: analysis
Issue: Orphaned SECTION_START (no SECTION_END found)
Strategy: remove
Action: Removed 847 characters of partial content

Before: 
  <!-- SECTION_START: analysis | hash:null -->
  ## Analysis
  
  Partial content here...
  [truncated]

After:
  Section 'analysis' reset to pending status

Backup created: report.md.backup.20240115-103045

Ready to regenerate: /stream.write analysis

/stream.finalize

Strip markers and validate completeness.

Usage: /stream.finalize <filepath> [--output <output-filepath>]

Workflow:

1. Run full integrity check
2. Verify all sections completed
3. Verify all hashes valid
4. Remove SECTION_START and SECTION_END markers
5. Remove YAML frontmatter stream metadata
6. Validate no incomplete markers remain
7. Write to output file (or overwrite in place)

Script:

python scripts/stream_cleanup.py report.md --output final_report.md

Pre-finalize validation:

Finalize Check: report.md

Sections:
  [x] introduction ✓
  [x] background ✓
  [x] analysis ✓
  [x] recommendations ✓
  [x] conclusion ✓

All sections complete: YES
All hashes valid: YES
Ready to finalize: YES

Finalizing...
Output written to: final_report.md
Markers removed: 10
Lines in final document: 1,247

If validation fails:

Finalize Check: report.md

⚠️  CANNOT FINALIZE - Issues detected:

  [ ] recommendations - PENDING (not written)
  [!] conclusion - CORRUPTED (hash mismatch)

Fix required before finalization:
1. /stream.write recommendations
2. /stream.repair conclusion

Recovery Scenarios

Scenario 1: Context limit hit mid-section

The section has SECTION_START but no SECTION_END:

<!-- SECTION_START: analysis | hash:null -->
## Analysis

Partial content here...
[truncated - no SECTION_END]

Recovery:

1. /stream.status --verify detects incomplete section
2. /stream.repair analysis removes partial content and preserves it as context
3. /stream.status shows the preserved context file with preview
4. /stream.write analysis continues from where interrupted (don't restart!)

Context Preservation: When repair runs, incomplete content is saved to a .context file:

report.analysis.context  # Contains the incomplete content

This allows Claude to:

• Review what was being written
• Continue coherently from where interrupted
• Not waste tokens regenerating already-written content

Scenario 2: Session ended between sections

All written sections have both markers:

<!-- SECTION_END: background | hash:e5f6g7h8 -->

Recovery:

1. /stream.resume finds next pending section
2. Continue with /stream.write

Scenario 3: Hash mismatch detected

START and END hashes don't match (content corrupted during write):

<!-- SECTION_START: analysis | hash:a1b2c3d4 -->
...
<!-- SECTION_END: analysis | hash:x9y8z7w6 -->

Recovery:

1. /stream.status --verify flags hash mismatch
2. /stream.repair analysis --strategy remove
3. /stream.write analysis regenerates

Scenario 4: Context compaction during heredoc append (what to avoid)

Problem: Using cat >> file << 'EOF' without streaming-output markers:

# DON'T DO THIS for long documents
cat >> spec.md << 'EOF'
## Section 12
Content...
EOF

If context compaction occurs, the heredoc may be split, causing:

• Partial content written
• No markers to detect corruption
• No way to identify resume point

Prevention: Always use streaming-output for documents >1000 lines.

B-SPEC Template

For B-SPEC documents, use the built-in template:

/stream.init b-spec-010-llm-integration.md --template bspec

This creates the standard 15-section structure:

stream_plan:
  version: "2.0"
  template: bspec
  sections:
    - id: overview
      status: pending
      hash: null
    - id: architecture
      status: pending
      hash: null
    - id: section-03  # Domain-specific
      status: pending
      hash: null
    - id: section-04
      status: pending
      hash: null
    - id: section-05
      status: pending
      hash: null
    - id: section-06
      status: pending
      hash: null
    - id: section-07
      status: pending
      hash: null
    - id: section-08
      status: pending
      hash: null
    - id: section-09
      status: pending
      hash: null
    - id: section-10
      status: pending
      hash: null
    - id: section-11
      status: pending
      hash: null
    - id: database-schema
      status: pending
      hash: null
    - id: api-specification
      status: pending
      hash: null
    - id: implementation-guide
      status: pending
      hash: null
    - id: appendices
      status: pending
      hash: null

Workflow Checklist

Copy and track progress:

Stream Progress: [document-name]
- [ ] Initialize file with section plan
- [ ] Write each section (one at a time):
  - [ ] Section 1: ___________
  - [ ] Section 2: ___________
  - [ ] Section 3: ___________
  - [ ] ...
- [ ] Run status --verify to check integrity
- [ ] Repair any corrupted sections
- [ ] Finalize to strip markers

Context-Aware Resume

When context compaction interrupts a section write, the system preserves your work:

Automatic Context Preservation

1. During repair: Incomplete content is saved to <file>.<section>.context
2. On status check: Context files are detected and previewed
3. When resuming: Review the context file to continue coherently

Resume Workflow

# 1. Check status and see context
python scripts/stream_status.py report.md

# Output includes:
# 📝 Context files (from previous incomplete writes):
#    analysis: report.analysis.context (2847 bytes)
#    Preview (last 500 chars):
#    | The normalization formula uses...
#    | This ensures that all criteria...

# 2. Read the full context if needed
cat report.analysis.context

# 3. Continue writing (DON'T restart from scratch)
# Your content should pick up where the context file ended

Important: Continue, Don't Restart

When a context file exists:

• DO: Read it, understand where you left off, continue from that point
• DON'T: Ignore it and regenerate the entire section from scratch

This saves significant tokens and maintains coherence.

Best Practices

1. Plan sections upfront: Define all sections before starting
2. One section at a time: Write, verify, then proceed
3. Check status often: Especially after interruptions or compaction
4. Keep sections reasonable: 500-2000 words per section works well
5. Save context: Don't hold generated content in memory; write immediately
6. Use --verify flag: Run /stream.status --verify after resuming
7. Never skip repair: If corruption is detected, repair before continuing
8. Use templates: For known document types like B-SPECs, use built-in templates
9. Review context files: When resuming, always check for and use .context files

Anti-Patterns to Avoid

Integration with Continuation Prompts

When creating continuation prompts for long document generation:

1. Explicitly state: "Use streaming-output skill for all document generation"
2. Include status command: "Run /stream.status <file> --verify to check progress"
3. Document section plan: List all sections and their status
4. Specify resume point: "Resume from section: X"

Example continuation prompt snippet:

## Document Generation Status

Using streaming-output skill for B-SPEC-010.

Current status:
- Sections 1-5: Complete ✓
- Section 6: Pending (resume here)
- Sections 7-15: Pending

Resume command: `/stream.resume b-spec-010.md`

Scripts Reference

Run any script with --help for detailed options.

Changelog

v2.1 (2026-01-03)

• Context preservation on repair: Incomplete content now saved to .context files
• Context-aware status: Status command shows context files with previews
• Resume guidance: Clear instructions to continue from context, not restart
• Added "Context-Aware Resume" section with workflow guidance
• Updated recovery scenarios to emphasize context preservation

v2.0 (2026-01-03)

• Added /stream.repair command for fixing corrupted sections
• Added hash-based integrity checking for section markers
• Added --verify flag to status command
• Added B-SPEC template (--template bspec)
• Added corruption detection (orphaned START, hash mismatch, empty sections)
• Added "Mandatory Use Cases" section
• Added "Anti-Patterns to Avoid" section
• Added "Integration with Continuation Prompts" guidance
• Improved recovery scenarios with specific commands
• Enhanced status output with integrity indicators

v1.0 (Original)

• Initial streaming output implementation
• Basic section markers
• Status and finalize commands