dream — KB memory consolidation
Overview
Dream is the maintenance pass that keeps a markdown knowledge base compact and
retrievable — the agent equivalent of sleep consolidating the day's memories. Run it when
the KB feels noisy (duplicates, contradictions, sprawl, stale facts), on a schedule, or
to forget specific notes. Everything is proposed as a diff first; nothing changes until
you approve (or you run --auto).
Dream is KB-agnostic and generic. It never assumes a fixed location.
Invocation
dream [<path>] — consolidate the KB at <path>. If omitted, use $KB_DIR; if that
is unset, ask for the path. Examples: dream ./docs/kb, dream $KB_DIR.
dream --auto [<path>] — non-interactive: apply merges and prunes silently, skip
contradictions (they need human judgment), guarded by a lock file.
dream forget "<query>" [<path>] — find notes matching a query and propose deletion
(same diff-for-approval flow as consolidation).
A note is any *.md file with YAML frontmatter. Map missing fields gracefully:
---
name: type-short-slug # equals the filename stem
description: one line, keyword-front-loaded
type: tech # user | feedback | project | reference | tech | ...
tags: [topic, topic]
sources: ["https://example.com/llms.txt"]
created: 2026-06-01
updated: 2026-06-13
pinned: false # optional — pinned notes are never pruned or merged
confidence: 0.8 # optional — 0..1; low + generic => prune candidate
---
- No
type => unknown. No created/updated => use file mtime. No confidence =>
neutral. No pinned => false.
The pass (run strictly in order)
Work in memory until the apply step; modify nothing early.
- Resolve the KB. Confirm
<path> holds memory/ (notes) and ideally MEMORY.md
(index), raw/inbox/ (captures), raw/ (source docs), and a state file. If the
layout differs, operate on whatever *.md files exist and adapt.
- Load. Read the index, every note,
raw/inbox/*, source docs, and state.
- Ingest inbox. Promote durable, general facts from
raw/inbox/* into memory/
(create or merge notes). Delete inbox files once distilled (they are ephemeral). Keep
raw/ source docs (immutable ground truth); record processed ones in state.
- Dedupe and merge (near-duplicates). Group notes by
type. Two notes are
near-duplicates when: same type; significant-noun overlap above ~60% (a proxy for
cosine similarity > 0.9); neither is pinned. Draft a merged note that is more
complete and specific than either, keeping the clearest, most recent phrasing.
Lossless of meaning, not of words.
- Contradictions. Two notes assert opposing facts on the same topic (for example
"deploy to ECS" vs "deploy to Vercel"). Proposed winner: more recent with higher
confidence. Present both for A/B/skip (interactive); skip in
--auto.
- Split. If a note holds multiple distinct facts, split into atomic notes (one fact
per file), renamed to
<type>-<slug>.md.
- Validate frontmatter. Every note has
name (equals filename stem), description,
type, tags, created, updated. Fix the <type>- filename prefix if the type
changed. No nested metadata: blocks.
- Refresh from sources. For notes with a
sources: URL whose updated: is stale
(older than ~30 days) or whose facts look outdated, fetch the source (prefer
llms.txt), update the note, bump updated:. Skip silently when offline.
- Compact. Trim each note to essential facts; drop filler. Aim for ~25 lines or
fewer.
- Prune candidates. A note is a prune candidate when any holds:
- its
type has a retention policy and it is older than the policy's days (compare
created/updated to today);
confidence is below 0.3 AND it has no project-unique info (no paths,
identifiers, or domain nouns);
- it is wrong, obsolete, or out of scope.
Never prune
pinned: true. Prefer refreshing (step 8) over deleting merely because
a note is old.
- Relink and retag. Fix
[[slug]] references; ensure no orphans (each note links
at least one other); merge tag sprawl into the controlled vocabulary; drop links to
deleted notes.
- Rebuild index. Regenerate
MEMORY.md from surviving notes, grouped by type,
one line each: [Title](file) — hook.
- Verify scope. Confirm no secrets, hosts, or confidential facts slipped in;
remove any that did.
- Stamp. Set
state.last_dream to today.
Retention policies (load in step 1)
Default per-type retention, in days (null means keep forever). Override via a
retention: map in the KB's state or config file.
feedback: 180
project: null
reference: null
tech: 365
user: null
unknown: 90
Diff report (interactive)
Print exactly this before any change. Omit any section that has zero items.
## dream — consolidation report
Merges (<N>):
[note:<file1>] + [note:<file2>] -> "<merged summary, ~80 chars>"
Conflicts (<N>):
[note:<fileA>] vs [note:<fileB>] — "<topic>" [A/B/skip]
Prune (<N>):
[note:<file>] — <type>, <age>d old
Inbox ingested (<N>), refreshed (<N>), split (<N>), relinked (<N>).
Proposed: <N> merges, <N> prunes, <N> conflicts. Apply? [Y/n]
If there are zero proposals, print Dream complete. No duplicate, contradictory, or stale notes found. and stop.
Resolve and apply (interactive)
- For each conflict, collect
A, B, or skip (empty input = skip).
- Final confirm:
Apply? [Y/n]. n or no prints Cancelled. No changes made. and
stops.
- On confirm, apply in this order:
- Merges — write the merged note (
<type>-<slug>.md; type and updated = today;
confidence = the higher of the two originals; a source: dream marker), delete
both originals, and rewrite inbound links to point at the merged slug.
- Contradictions — delete the loser of each resolved A/B pair; skipped pairs stay
untouched.
- Prunes — delete the note and its index line.
- Split, refresh, compact, relink, index rebuild — apply the file writes.
- Stamp state, and run the KB's sync step only if one exists and only after the
user confirms — never auto-push.
Auto mode (--auto)
- Concurrency guard. Before working, check
${TMPDIR:-/tmp}/dream_auto.lock. If it
exists and is less than 10 minutes old, print
[dream --auto] another run in progress — skipping. and stop. Otherwise write the lock
(current timestamp); delete it on every exit path.
- Run load and analyze (steps 1–3). Apply merges and prunes silently (no diff, no
prompt). Skip contradictions (human judgment). Apply split, refresh, compact, relink,
index rebuild, and stamp.
- Print:
[dream --auto] kb=<path> merged=<N> pruned=<N> conflicts_skipped=<N>.
- If contradictions were skipped, leave a single reminder note so the human runs
interactive
dream to resolve them. First search for an existing dream-auto reminder
(by source: dream-auto); skip storing if one already exists.
forget
dream forget "<query>" [<path>] searches notes by keyword and semantic match, lists
them, and proposes deletion as a diff:
[note:<file>] — "<title>" [Y/n]
Apply only approved deletions (and their index lines). Pinned notes are listed but not
deleted unless explicitly confirmed per note.
Principles
- Lossless of meaning — compaction removes words and redundancy, never facts.
- Atomic — one fact per file after the pass.
- Retrieval-first — every
description lets an agent judge relevance from the index
alone; every note is reachable via tags and [[links]].
- Conservative deletes — prune only what is wrong, redundant, or out of scope; refresh
rather than delete just because old.
- Idempotent — running twice in a row changes nothing the second time.
- No surprises — diff before write; never auto-push or auto-sync without approval.
Checklist
Resources
- If the KB ships its own
DREAM.md or AGENTS.md, read it first and align with its note
format, controlled vocabulary, and sync steps.
1---
2name: dream
3description: Consolidate a markdown knowledge base — merge near-duplicates, flag contradictions, prune stale notes, ingest inbox, refresh from sources, split, and rebuild the index. Use when a KB is noisy or stale, on a schedule, or to forget notes. Diff-for-approval; supports --auto.
4metadata:
5 source: https://skillmd.com/skills/duyet/dream
6---
7
8# dream — KB memory consolidation
9
10## Overview
11
12Dream is the maintenance pass that keeps a markdown knowledge base compact and
13retrievable — the agent equivalent of sleep consolidating the day's memories. Run it when
14the KB feels noisy (duplicates, contradictions, sprawl, stale facts), on a schedule, or
15to forget specific notes. Everything is proposed as a diff first; nothing changes until
16you approve (or you run `--auto`).
17
18Dream is KB-agnostic and generic. It never assumes a fixed location.
19
20## Invocation
21
22- `dream [<path>]` — consolidate the KB at `<path>`. If omitted, use `$KB_DIR`; if that
23 is unset, ask for the path. Examples: `dream ./docs/kb`, `dream $KB_DIR`.
24- `dream --auto [<path>]` — non-interactive: apply merges and prunes silently, skip
25 contradictions (they need human judgment), guarded by a lock file.
26- `dream forget "<query>" [<path>]` — find notes matching a query and propose deletion
27 (same diff-for-approval flow as consolidation).
28
29A note is any `*.md` file with YAML frontmatter. Map missing fields gracefully:
30
31```yaml
32---
33name: type-short-slug # equals the filename stem
34description: one line, keyword-front-loaded
35type: tech # user | feedback | project | reference | tech | ...
36tags: [topic, topic]
37sources: ["https://example.com/llms.txt"]
38created: 2026-06-01
39updated: 2026-06-13
40pinned: false # optional — pinned notes are never pruned or merged
41confidence: 0.8 # optional — 0..1; low + generic => prune candidate
42---
43```
44
45- No `type` => `unknown`. No `created`/`updated` => use file mtime. No `confidence` =>
46 neutral. No `pinned` => false.
47
48## The pass (run strictly in order)
49
50Work in memory until the apply step; modify nothing early.
51
521. **Resolve the KB.** Confirm `<path>` holds `memory/` (notes) and ideally `MEMORY.md`
53 (index), `raw/inbox/` (captures), `raw/` (source docs), and a state file. If the
54 layout differs, operate on whatever `*.md` files exist and adapt.
552. **Load.** Read the index, every note, `raw/inbox/*`, source docs, and state.
563. **Ingest inbox.** Promote durable, general facts from `raw/inbox/*` into `memory/`
57 (create or merge notes). Delete inbox files once distilled (they are ephemeral). Keep
58 `raw/` source docs (immutable ground truth); record processed ones in state.
594. **Dedupe and merge (near-duplicates).** Group notes by `type`. Two notes are
60 near-duplicates when: same `type`; significant-noun overlap above ~60% (a proxy for
61 cosine similarity > 0.9); neither is `pinned`. Draft a merged note that is more
62 complete and specific than either, keeping the clearest, most recent phrasing.
63 Lossless of meaning, not of words.
645. **Contradictions.** Two notes assert opposing facts on the same topic (for example
65 "deploy to ECS" vs "deploy to Vercel"). Proposed winner: more recent with higher
66 confidence. Present both for A/B/skip (interactive); skip in `--auto`.
676. **Split.** If a note holds multiple distinct facts, split into atomic notes (one fact
68 per file), renamed to `<type>-<slug>.md`.
697. **Validate frontmatter.** Every note has `name` (equals filename stem), `description`,
70 `type`, `tags`, `created`, `updated`. Fix the `<type>-` filename prefix if the type
71 changed. No nested `metadata:` blocks.
728. **Refresh from sources.** For notes with a `sources:` URL whose `updated:` is stale
73 (older than ~30 days) or whose facts look outdated, fetch the source (prefer
74 `llms.txt`), update the note, bump `updated:`. Skip silently when offline.
759. **Compact.** Trim each note to essential facts; drop filler. Aim for ~25 lines or
76 fewer.
7710. **Prune candidates.** A note is a prune candidate when any holds:
78 - its `type` has a retention policy and it is older than the policy's days (compare
79 `created`/`updated` to today);
80 - `confidence` is below 0.3 AND it has no project-unique info (no paths,
81 identifiers, or domain nouns);
82 - it is wrong, obsolete, or out of scope.
83 Never prune `pinned: true`. Prefer refreshing (step 8) over deleting merely because
84 a note is old.
8511. **Relink and retag.** Fix `[[slug]]` references; ensure no orphans (each note links
86 at least one other); merge tag sprawl into the controlled vocabulary; drop links to
87 deleted notes.
8812. **Rebuild index.** Regenerate `MEMORY.md` from surviving notes, grouped by `type`,
89 one line each: `[Title](file) — hook`.
9013. **Verify scope.** Confirm no secrets, hosts, or confidential facts slipped in;
91 remove any that did.
9214. **Stamp.** Set `state.last_dream` to today.
93
94## Retention policies (load in step 1)
95
96Default per-type retention, in days (`null` means keep forever). Override via a
97`retention:` map in the KB's state or config file.
98
99- `feedback`: 180
100- `project`: null
101- `reference`: null
102- `tech`: 365
103- `user`: null
104- `unknown`: 90
105
106## Diff report (interactive)
107
108Print exactly this before any change. Omit any section that has zero items.
109
110```text
111## dream — consolidation report
112
113Merges (<N>):
114 [note:<file1>] + [note:<file2>] -> "<merged summary, ~80 chars>"
115
116Conflicts (<N>):
117 [note:<fileA>] vs [note:<fileB>] — "<topic>" [A/B/skip]
118
119Prune (<N>):
120 [note:<file>] — <type>, <age>d old
121
122Inbox ingested (<N>), refreshed (<N>), split (<N>), relinked (<N>).
123
124Proposed: <N> merges, <N> prunes, <N> conflicts. Apply? [Y/n]
125```
126
127If there are zero proposals, print `Dream complete. No duplicate, contradictory, or stale
128notes found.` and stop.
129
130## Resolve and apply (interactive)
131
132- For each conflict, collect `A`, `B`, or `skip` (empty input = skip).
133- Final confirm: `Apply? [Y/n]`. `n` or `no` prints `Cancelled. No changes made.` and
134 stops.
135- On confirm, apply in this order:
136 1. **Merges** — write the merged note (`<type>-<slug>.md`; `type` and `updated` = today;
137 `confidence` = the higher of the two originals; a `source: dream` marker), delete
138 both originals, and rewrite inbound links to point at the merged slug.
139 2. **Contradictions** — delete the loser of each resolved A/B pair; skipped pairs stay
140 untouched.
141 3. **Prunes** — delete the note and its index line.
142 4. **Split, refresh, compact, relink, index rebuild** — apply the file writes.
143 5. **Stamp** state, and run the KB's sync step only if one exists and only after the
144 user confirms — never auto-push.
145
146## Auto mode (`--auto`)
147
148- **Concurrency guard.** Before working, check `${TMPDIR:-/tmp}/dream_auto.lock`. If it
149 exists and is less than 10 minutes old, print
150 `[dream --auto] another run in progress — skipping.` and stop. Otherwise write the lock
151 (current timestamp); delete it on every exit path.
152- Run load and analyze (steps 1–3). Apply merges and prunes silently (no diff, no
153 prompt). Skip contradictions (human judgment). Apply split, refresh, compact, relink,
154 index rebuild, and stamp.
155- Print: `[dream --auto] kb=<path> merged=<N> pruned=<N> conflicts_skipped=<N>`.
156- If contradictions were skipped, leave a single reminder note so the human runs
157 interactive `dream` to resolve them. First search for an existing `dream-auto` reminder
158 (by `source: dream-auto`); skip storing if one already exists.
159
160## forget
161
162`dream forget "<query>" [<path>]` searches notes by keyword and semantic match, lists
163them, and proposes deletion as a diff:
164
165```text
166[note:<file>] — "<title>" [Y/n]
167```
168
169Apply only approved deletions (and their index lines). Pinned notes are listed but not
170deleted unless explicitly confirmed per note.
171
172## Principles
173
174- **Lossless of meaning** — compaction removes words and redundancy, never facts.
175- **Atomic** — one fact per file after the pass.
176- **Retrieval-first** — every `description` lets an agent judge relevance from the index
177 alone; every note is reachable via tags and `[[links]]`.
178- **Conservative deletes** — prune only what is wrong, redundant, or out of scope; refresh
179 rather than delete just because old.
180- **Idempotent** — running twice in a row changes nothing the second time.
181- **No surprises** — diff before write; never auto-push or auto-sync without approval.
182
183## Checklist
184
185- [ ] Resolved KB path from the argument or `$KB_DIR`; never assumed a fixed location.
186- [ ] Held everything in memory; no writes before the diff.
187- [ ] Inbox ingested and deleted; source docs kept.
188- [ ] Near-duplicates merged (same type, >60% noun overlap, not pinned).
189- [ ] Contradictions resolved (A/B) or skipped (`--auto`).
190- [ ] Stale notes refreshed from sources before pruning; pinned notes untouched.
191- [ ] Index rebuilt; no orphans; no broken `[[links]]`.
192- [ ] Scope verified (no secrets, hosts, or confidential facts).
193- [ ] State stamped; lock removed (`--auto`).
194
195## Resources
196
197- If the KB ships its own `DREAM.md` or `AGENTS.md`, read it first and align with its note
198 format, controlled vocabulary, and sync steps.