Streaming Protocols

Streaming Protocols for AI Systems

Iron Law

Pick one option from each of the three layers (Transport / Framing / Message). The "which protocol?" question answers itself.

Most streaming confusion comes from treating options at different layers as alternatives. They compose — they are not competitors.

1. The Three-Layer Mental Model

┌─────────────────────────────────────────────────────────┐
│  MESSAGE LAYER — what each message means                │
│  JSON-RPC · A2UI envelope · OpenAI delta · plain JSON   │
├─────────────────────────────────────────────────────────┤
│  FRAMING LAYER — how messages are delimited             │
│  SSE events · NDJSON lines · HTTP chunks · WS frames    │
├─────────────────────────────────────────────────────────┤
│  TRANSPORT LAYER — the connection itself                │
│  HTTP · HTTP Streaming · WebSocket · stdio              │
└─────────────────────────────────────────────────────────┘

Transport opens the pipe.
Framing chops the pipe into messages.
Message format gives each message meaning.

The most common real-world stack:

HTTP Streaming (transport)
  └── SSE (framing)
        └── JSON-RPC or provider-specific JSON (message)

For tool/agent IPC:

stdio (transport)
  └── newline-delimited JSON (framing)
        └── JSON-RPC 2.0 (message)

2. SSE vs HTTP Streaming — The Core Confusion

SSE is not a competitor to HTTP Streaming. SSE is one kind of HTTP Streaming.

HTTP Streaming  (the behavior: server sends response gradually)
    │
    ├── SSE                 ← text/event-stream, blank-line framed
    ├── NDJSON              ← one JSON object per line
    ├── Chunked HTML        ← progressive page rendering
    ├── Raw binary chunks   ← video/audio, file downloads
    └── Custom formats      ← gRPC-Web, protobuf streams

Asking "SSE vs HTTP Streaming" is like asking "JSON vs HTTP body" — JSON is one of the things you can put in a body.

Why people conflate them:

1. In browsers, EventSource is the only ergonomic way to consume HTTP streams. Everything else requires fetch() + ReadableStream + manual byte parsing.
2. ChatGPT, Claude, Perplexity all use SSE — so "streaming response" colloquially means "SSE response."
3. MCP's transport naming confused everyone: the old transport was "HTTP+SSE"; the new one is "Streamable HTTP" — these sound like alternatives when they're not.

When to pick which framing:

3. Bidirectionality — Half-Duplex vs Full-Duplex

The crisp distinction:

HTTP Streaming = one-direction-at-a-time per request (walkie-talkie)
WebSocket      = both-directions-simultaneously on one connection (phone call)

Why ChatGPT's "stop generating" button closes the connection: once the POST is sent, the client is in receive-only mode. There's no upstream channel to send a "stop" message on. The only option is to drop the connection.

Pick HTTP Streaming (SSE) when:

• Data flow is dominantly server → client
• You need to traverse corporate proxies and CDNs reliably
• You want browser auto-reconnect and resumability for free
• You're streaming LLM tokens, agent progress, A2UI updates

Pick WebSocket when:

• Genuinely simultaneous bidirectional flow (voice, multiplayer, collab editing)
• Frequent small messages in both directions
• Server initiates messages as often as the client does

Practical reality for AI apps: 95% of the time, SSE is the right choice. WebSocket earns its weight for voice and collaborative editing — not for token streaming.

4. MCP Bidirectionality Pattern (How HTTP Fakes Full-Duplex)

MCP needs bidirectional communication (clients call tools; servers call back via sampling/createMessage, roots/list, elicitation). It achieves this over half-duplex HTTP using two simultaneous requests:

CLIENT                                MCP SERVER
  │                                     │
  │── POST /mcp (tools/call) ──────────►│
  │◄── HTTP 200, SSE response ──────────│  ← Stream A: responds to client POST
  │◄── data: {progress notification}    │
  │◄── data: {final result}             │
  │                                     │
  │── GET  /mcp (Accept: SSE) ─────────►│
  │◄── HTTP 200, SSE response ──────────│  ← Stream B: long-lived GET for
  │◄── data: {server→client request}    │    server-initiated messages
  │                                     │
  │── POST /mcp (response to above) ───►│  ← Client replies via new POST
  │◄── HTTP 200 ────────────────────────│

"Bidirectional messaging" in MCP = messages flow in both directions across two streams, not both directions on a single TCP stream.

Two headers that carry the operational glue:

SSE deprecation note: SSE was deprecated as a standalone MCP transport (March 2025, replaced by Streamable HTTP). SSE itself is alive and well for LLM streaming, dashboards, and all browser-facing live updates.

5. Real-World Wire Formats

OpenAI / Compatible APIs (SSE, bare JSON)

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","choices":[{"delta":{"content":"The"},"index":0}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","choices":[{"delta":{"content":" answer"},"index":0}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","choices":[{"delta":{},"finish_reason":"stop","index":0}]}

data: [DONE]

• Bare JSON (not JSON-RPC) — request/response pairing is implicit in the HTTP exchange
• [DONE] sentinel is OpenAI's convention, not part of the SSE spec
• Client extracts choices[0].delta.content and appends to rendered message

Anthropic (SSE, typed events)

event: message_start
data: {"type":"message_start","message":{"id":"msg_01...","role":"assistant"}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"The answer"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_stop
data: {"type":"message_stop"}

• event: field carries semantic type — clients dispatch on type rather than inspecting payload shape
• Makes tool use, thinking blocks, and multi-modal content cleaner to handle
• Mid-stream errors arrive as event: error with HTTP status still 200

MCP (JSON-RPC over SSE or newline-delimited)

event: message
data: {"jsonrpc":"2.0","method":"notifications/progress","params":{"progressToken":"t1","progress":30}}

event: message
data: {"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"done"}]}}

A2UI over SSE

event: a2ui
data: {"version":"v0.9","createSurface":{"surfaceId":"ticket_1","catalogId":"..."}}

event: a2ui
data: {"version":"v0.9","updateDataModel":{"surfaceId":"ticket_1","path":"/status","value":"Open"}}

A2UI over NDJSON (replay / eval / debugging)

{"version":"v0.9","createSurface":{"surfaceId":"ticket_1","catalogId":"..."}}
{"version":"v0.9","updateDataModel":{"surfaceId":"ticket_1","path":"/status","value":"Open"}}
{"version":"v0.9","updateComponents":{"surfaceId":"ticket_1","components":[...]}}

6. Production Pitfalls Checklist

❌ Proxy Buffering (Silent Stream Killer)

Nginx, CloudFront, and most CDNs buffer responses by default. Your "streaming" tokens arrive in one batch.

Fix:

location /stream {
  proxy_buffering off;
  add_header X-Accel-Buffering no;
}

Also set X-Accel-Buffering: no as a response header from your app server.

❌ gzip / Brotli Breaks SSE

Compression middleware accumulates bytes before compressing — turns smooth token flow into chunky bursts.

Fix: Disable compression for text/event-stream responses. The bandwidth saving is not worth the latency cost.

// Express example
app.use(compression({
  filter: (req, res) => {
    if (res.getHeader('Content-Type') === 'text/event-stream') return false;
    return compression.filter(req, res);
  },
}));

❌ SSE Connection Timeouts on Long Agent Runs

Long agent runs exceed default proxy timeouts (often 60s). Connection drops mid-stream.

Fix: Send periodic keepalive comments:

: keepalive

(A colon-prefixed line is an SSE comment — ignored by clients, but prevents timeout.)

❌ Mobile Reconnection Without Last-Event-ID

iOS kills HTTP connections when the app backgrounds. Without Last-Event-ID, the stream restarts from scratch on reconnect.

Fix:

1. Include id: field on every SSE event
2. Buffer events server-side for ≥60 seconds after sending
3. On reconnect, client sends Last-Event-ID header; server replays from that point

// Client: use exponential backoff — default EventSource reconnect is too aggressive
const es = new EventSource('/stream');
let retryDelay = 1000;
es.onerror = () => {
  setTimeout(() => reconnect(), retryDelay);
  retryDelay = Math.min(retryDelay * 2, 30000);
};

4. Surface "reconnecting..." state to the user — silent staleness is worse than a visible indicator
5. For long agent runs (>10 min backgrounded): provide a polling fallback that returns current state

❌ Missing traceparent Propagation

A single user action may generate 30+ network calls. Without a correlation ID, cross-service debugging is impossible.

Fix: Generate traceparent (W3C Trace Context) at the edge; forward it on every outbound call.

User action
  └─► Backend generates traceparent: 00-{trace_id}-{span_id}-01
        ├─► POST to LLM API — Header: traceparent
        ├─► POST to MCP server — Header: traceparent, Mcp-Session-Id
        └─► SSE response to client — echo traceparent back

Every modern observability tool (Langfuse, Datadog, Honeycomb, Grafana Tempo) understands traceparent. Include it in NDJSON logs so you can join logs across services after the fact.

❌ Conflating JSON-RPC id and SSE id

They are unrelated and live at different layers.

❌ Treating Streamable HTTP and SSE as Competitors

Streamable HTTP (MCP) is a transport pattern. SSE is one of the response formats Streamable HTTP can return. They compose.

❌ WebSocket "Because It's Realtime"

If data flow is server→client only (which most LLM streaming is), WebSocket adds operational complexity without benefit. SSE is simpler, proxy-friendly, and auto-reconnecting. Reach for WebSocket only when you genuinely need bidirectional realtime.

7. Decision Guide

8. NDJSON's Real-World Niche

NDJSON rarely appears in the live request path (SSE wins there). It dominates in three places:

1. stdio IPC — MCP local servers, language servers, CLI tools. Newline framing is OS-pipe-friendly.
2. Batch / bulk APIs — OpenAI Batch API, Anthropic Batch API return NDJSON. Common Crawl, eval traces, log shipping (Vector, Fluent Bit) all favor NDJSON.
3. Replay and eval — Recording an agent run as NDJSON gives a perfect, line-addressable trace. Langfuse, Braintrust, W&B Weave ingest these directly.

SSE        → live, server→client, browser-facing
NDJSON     → batch, logs, replay, stdio, server-to-server
WebSocket  → genuinely bidirectional realtime (voice, collab editing)
Plain JSON → one-shot response, no streaming needed

Cheat Sheet

Three rules that catch 90% of bugs:

1. Don't compress SSE responses — it buffers your stream into chunky bursts.
2. Echo Mcp-Session-Id on every MCP request after init, or session state silently breaks.
3. Propagate traceparent across every layer, or you'll never debug a multi-hop agent failure.

The four real-world stacks: