Streaming Output Reader

Streaming Output Reader

Long outputs kill conversations. A 50,000-line log dump will fill any context window in one read. The reflex is to tail the file or read a small slice — but that often misses the earliest lines (errors at the start of a long run) and requires multiple round trips to get a complete picture.

This Skill defines a single-pass read protocol: read in bounded chunks, keep a cumulative summary, and stop when you have enough to act on.

When to use

Activate when any of these is true:

• A tool returns output that might be longer than ~3000 tokens (the tool-output-budget threshold). Better to be cautious: read in chunks from the start.
• The tool offers an explicit streaming API (SSE, WebSocket chunks, tail -f with follow-mode, log subscription) and you are not using it.
• You are about to read a file you do not control the size of (build logs, test logs, process stdout, JSONL files).
• A previous read returned truncation / "output cut off" / "use offset to read more".

When NOT to use

• The output is known to be small (< 100 lines). Just read or cat it in one go.
• The output is structured (JSON / CSV) and you need the whole thing to parse. Read it once with a guard ("first 3000 lines"), then if you need more, do a second targeted read.
• You are polling for a specific event ("did the file appear yet?"). That is a different pattern (background-task Skill + read with a wait).

Process

1. Estimate before reading. If you do not know the size, use a low-cost probe (e.g. wc -l file.log, ls -la file.log, or du -h) to learn the byte / line count before reading content. This prevents "oh, 200 MB" surprises.
2. Pick a chunk policy. Default:
   • head -N for the first N lines (where N ≈ 200) — captures start-of-run errors and configuration.
   • tail -N for the last N lines (where N ≈ 50) — captures end-of-run status, exit messages, final stack trace.
   • grep -B 2 -A 2 PATTERN for targeted search — captures context around a known error or marker.
3. Read once, in chunks, do not loop. Three reads in a single response is acceptable; ten is not. After three reads, stop and synthesize — re-reading the same file in different windows is a sign the data is too noisy for the current approach, and the right answer is ask-user or switch to a different file.
4. Maintain a cumulative summary as you read. The summary is what the user sees; the raw chunks are scaffolding. After each chunk, append a one-line takeaway to the summary. The summary fits in your context; the raw output does not.
5. Stop when you have enough to act on. "Enough" means:
   • You can state the file's purpose and current state in one sentence.
   • You can identify any errors or anomalies.
   • You can answer the user's original question, OR you know what you would need to read next to answer it (and the next read is targeted, not another dump).
6. If you cannot stop, surface to the user with the cumulative summary, the question, and the trade-off (read more context? read a different file? ask the user?).

Output contract

The user sees, in this order:

• One-line estimate of the file / stream size (if probed).

• A cumulative summary in this shape:

  Stream summary: <one-line purpose>
  Chunks read: <N> (head 200 / tail 50 / grep 1)
  Key findings:
  - <line N>: <finding>
  - <line M>: <finding>
  Next step: <what to do with the findings, or what to read next if needed>
  

• (If you stopped because of step 6) the question and trade-off, not a silent loop.

Example

> wc -l /var/log/app/today.log
4287 /var/log/app/today.log

> head -200 /var/log/app/today.log
[200 lines read; key entries captured below]

> grep -B 2 -A 5 "ERROR" /var/log/app/today.log | tail -50
[50 lines of error context; summary below]

Stream summary: application startup with a 10-minute error burst at 14:23
Chunks read: 3 (head 200 / tail 50 / grep 1)
Key findings:
- 14:23:00 INFO service starting
- 14:23:14 ERROR connect ECONNREFUSED 10.0.0.5:5432 (database)
- 14:23:14 ERROR retry 1/3 ...
- 14:23:23 ERROR retry 3/3 exhausted; service degraded
- 14:33:00 WARN service continuing in degraded mode (DB read-only)

Next step: investigate why 10.0.0.5:5432 was unreachable at 14:23; the
service is now in a degraded state and the user may need to restart.

Counter-example (read too much, lost the thread):

> read /var/log/app/today.log
[4000 lines, ~50K tokens; everything from the read is now in context]

> read /var/log/app/today.log
[duplicate, same 4000 lines]

> read /var/log/app/today.log
[duplicate, same 4000 lines — context is now full of redundant log]

The counter-example is the failure mode this Skill prevents. Three reads, no synthesis, context full, no actionable conclusion.

Common pitfalls

• Do not read an unbounded file in one call. Use head, tail, or grep first.
• Do not loop reads. Three is a hard cap. After three, you owe the user a synthesis, not a fourth read.
• Do not skip the summary. The raw chunks are scaffolding; the summary is the product. Without a summary, the user has to re-read everything you read.
• Do not guess from a single chunk. If the file is structured (timestamps, log levels), use grep to anchor on the structure, not just head/tail.
• Do not re-read the same range. If head -200 did not show what you needed, do not read head -200 again; read grep PATTERN or sed -n '200,400p'.
• Do not stream to the user's chat verbatim. The user wants the synthesis, not the raw bytes. Stream is for you; summary is for them.

Verification checklist

• Did you estimate size before reading (if size was unknown)?
• Did you use a chunk policy (head / tail / grep) rather than a single read?
• Did you write a cumulative summary as you went, not after?
• Did you stop after at most 3 reads, even if you did not have the answer?
• Is the summary one-line purpose + findings + next step, not raw output?
• Did you surface to the user (with the question) if you could not stop on your own?
• Did you avoid re-reading the same range?