Chunked Streaming

Chunked File Streaming

Transfer large files by splitting them into fixed-size chunks and streaming over TCP. Avoids loading the entire file into memory.

Architecture

Sender                          Receiver
  │                               │
  ├── META (name, size, hash)   ►│
  ├── CHUNK (1MB seq=0) ───────►│
  ├── CHUNK (1MB seq=1) ───────►│
  ├── CHUNK (1MB seq=2) ───────►│
  ├── ...                       │
  ├── DONE (final_hash) ───────►│
  │                               │
  │  Verify hash on receive       │

Python Implementation

Sender

import hashlib
import os

CHUNK_SIZE = 1024 * 1024  # 1 MB

def stream_file(filepath, send_chunk_fn):
    total = os.path.getsize(filepath)
    sha256 = hashlib.sha256()
    seq = 0

    with open(filepath, "rb") as f:
        while True:
            chunk = f.read(CHUNK_SIZE)
            if not chunk:
                break
            sha256.update(chunk)
            send_chunk_fn(chunk, seq, total)
            seq += 1

    return sha256.hexdigest()

Receiver

def receive_file(filepath, total_size, receive_chunk_fn):
    sha256 = hashlib.sha256()
    received = 0

    with open(filepath, "wb") as f:
        while received < total_size:
            chunk = receive_chunk_fn()
            if not chunk:
                break
            f.write(chunk)
            sha256.update(chunk)
            received += len(chunk)

    return sha256.hexdigest()

Protocol Format

Wrap each chunk in a length-prefixed message:

[4 bytes: total message length]
[message type: "META" | "CHUNK" | "DONE" | "ERROR"]
[payload]

For CHUNK messages:

[4 bytes: sequence number]
[4 bytes: chunk size]
[N bytes: chunk data]

Progress Tracking

Report progress every N chunks or every 500ms:

progress = received / total * 100
speed = bytes_in_last_second / 1024 / 1024  # MB/s
eta = (total - received) / speed  # seconds

Resumable Transfers

To support resume:

1. Receiver tells sender how many bytes it already has
2. Sender seeks to that position: f.seek(resume_offset)
3. Both sides continue from the offset

RECEIVER → RESUME_REQ(offset=524288)
SENDER   → META(..., resume_from=524288)
SENDER   → CHUNK(seq=1, data=...)

Verification

• SHA-256 — full-file hash sent in DONE message, verified by receiver
• Per-chunk CRC32 — optional, adds overhead but catches corruption early
• Compare hash — abort on mismatch

Constants