# Decide Streaming

> When to stream vs chunk vs load all. Memory vs complexity.

- Skill: `attac-t/decide-streaming` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add attac-t/decide-streaming`
- Raw SKILL.md: https://api.skillmd.com/api/skills/attac-t/decide-streaming/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: attac-t (https://skillmd.com/u/attac-t)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/attac-t/decide-streaming

---


# Skill: Streaming vs Chunking

> "Memory is cheap until it isn't."

## The Decision

**Load All when:**
- Dataset fits comfortably in memory (<10K records)
- Need random access or multiple passes
- Simplicity matters more than memory

**Chunk when:**
- Processing in batches (1K-10K per batch)
- Database operations (insert/update batches)
- Can tolerate batch boundaries

**Stream when:**
- Dataset size unknown or very large (100K+)
- Processing row-by-row is natural
- Memory is constrained

## The Heuristic

Ask: *"What happens if the dataset 10x in size?"*

## The Quick Test

| Ask | Answer | Use |
|-----|--------|-----|
| Will it always be <1K records? | Yes | Load all |
| Processing in batches of N? | Yes | Chunk |
| Could be millions of records? | Yes | Stream |
| Need all data at once for calculation? | Yes | Load all |

## Real-World Examples

See [examples.md](examples.md).