gotcontext-memory — streaming digests
Companion to gotcontext-memory-hitl-honesty. Load when the problem is scale, not HITL policy.
Why this exists
Measured corpus: ~9.6 GB / 11k+ Claude transcripts, including a ~2.3 GB file that
readFile rejects (> 2 GiB). Accumulating full Transcript[] cannot work. Dream must
stream into bounded digests and reason only over digests.
Rules
- Stream, don’t slurp —
digestTranscriptFile/digestRootsinsrc/dream/digest.ts. - Truncated ≠ malformed — byte ceiling →
truncated: true; parse/unreadable →malformed. - Counts vs samples —
DIGEST_SIGNAL_CAPbounds arrays; counts always increment. - Stratified window —
--max-sessions(default 400) viaselectDigests: ≈2/3 newest + evenly sampled older strata (session clock). Newest-N alone collapses calendar span. - Concurrent digestion — bounded worker pool (default 8); modest I/O-bound speedup.
.vscdbon the path — enumerate with*.jsonl;digestVscdbFile+ sharedclassifyText(BL-DRM-016 closed 2026-08-10). Unreadable.vscdb→ malformed, not fatal.- Prove on real size — oversized / truncated fixtures before claiming “works.”
Do not
- Reintroduce whole-file
readFilefor dream corpus. - Report only
malformedwhen the failure was a size ceiling. - Re-open a silent harness drop without HONESTY + backlog.
Authority
- Lessons L15, L18, L22 —
docs/LESSONS_2026-08-09.md docs/HONESTY.md,docs/features/dream.md,docs/guides/rebuild-from-scratch.md- Tests:
test/digest.test.ts,test/digest-vscdb.test.ts,test/digest-window.test.ts
Verify
cd gotcontext-memory && npm test -- test/digest.test.ts test/digest-vscdb.test.ts test/digest-window.test.ts