Background memory consolidation cycle. Scans memory files, finds stale/duplicate/conflicting entries, consolidates, synthesizes cross-session insights, builds an injection-ready payload for next session start, and writes a dated dream report.
When to invoke
- User says "run dream", "consolidate memories", "clean up memories", "memory maintenance", "deduplicate memories"
- Cron job at 2 AM nightly via wrapper script:
scripts/auto-dream-cron.sh --execute
- Manual trigger for testing:
./scripts/auto-dream-cron.sh (dry-run by default)
Reference Loading Table
| Signal |
Load These Files |
Why |
| Debugging failed cron run, silent failure, empty log, wrong exit code |
headless-cron-patterns.md |
Routes to the matching deep reference |
Setting up or modifying wrapper script (flock, --permission-mode, envsubst, PIPESTATUS) |
headless-cron-patterns.md |
Routes to the matching deep reference |
Budget cap, --max-budget-usd, unattended Claude invocation |
headless-cron-patterns.md |
Routes to the matching deep reference |
| Writing, updating, or archiving memory files |
memory-file-operations.md |
Routes to the matching deep reference |
Updating MEMORY.md index, atomic write, .tmp rename |
memory-file-operations.md |
Routes to the matching deep reference |
| Staleness detection, duplicate merging, conflict flagging |
memory-file-operations.md |
Routes to the matching deep reference |
YAML frontmatter structure, merged_from, memory file format |
memory-file-operations.md |
Routes to the matching deep reference |
| Testing the dream cycle safely, dry-run validation, output file verification |
dream-cycle-testing.md |
Routes to the matching deep reference |
| Inspecting graduation candidates, snapshot testing, PIPESTATUS in test wrappers |
dream-cycle-testing.md |
Routes to the matching deep reference |
| Reading and interpreting cron run logs, detecting silent failures |
logging-patterns.md |
Routes to the matching deep reference |
| Log rotation, log directory structure, phase completion markers in logs |
logging-patterns.md |
Routes to the matching deep reference |
last-dream.md stale, missing injection payload, cron log empty |
logging-patterns.md |
Routes to the matching deep reference |
| Concurrent dream runs, lockfile already held, duplicate cron invocations |
concurrency.md |
Routes to the matching deep reference |
MEMORY.md.tmp left behind, partial write recovery, atomic rename failure |
concurrency.md |
Routes to the matching deep reference |
database is locked, SQLite WAL mode, busy_timeout, concurrent DB access |
concurrency.md |
Routes to the matching deep reference |
local changes would be overwritten, git stash before GRADUATE branch switch |
concurrency.md |
Routes to the matching deep reference |
Instructions
When invoked interactively (not via cron), read skills/meta/auto-dream/dream-prompt.md and execute its phases directly. The prompt is self-contained — it describes the full seven-phase cycle including safety constraints, file paths, and output formats.
For cron invocation: the dream prompt is passed directly to claude -p and runs as a standalone headless session with no CLAUDE.md, no hooks, no project context. All instructions are embedded in the prompt.
Phases
- SCAN — Read all memory files, query learning.db sessions (last 7 days), read recent git log. Write scan document to
~/.claude/state/dream-scan-{date}.md.
- ANALYZE — Identify stale, duplicate, conflicting memories and cross-session patterns. Write analysis to
~/.claude/state/dream-analysis-{date}.md.
- CONSOLIDATE — Apply consolidation actions (max 5 changes). Archive stale/merged files, update MEMORY.md atomically.
- SYNTHESIZE — Create insight memories from cross-session patterns (max 2 new memories per cycle).
- GRADUATE — Promote mature learning DB entries (confidence >= 0.9, 3+ observations) into agent/skill files as permanent failure modes. Commits on
dream/graduate-YYYY-MM-DD branch for human review. Max 3 per cycle. (ADR-159)
- SELECT — Build injection-ready payload for session start. Write to
~/.claude/state/dream-injection-{project-hash}.md.
- REPORT — Write dream summary to
~/.claude/state/last-dream.md.
Safety constraints (always enforced)
- Never delete files — archive to
memory/archive/, never rm
- Write the REPORT before executing any CONSOLIDATE filesystem operations
- Maximum 5 memory changes per cycle — excess items deferred to next cycle
- Flag conflicts for human review, never auto-resolve
- Preserve YAML frontmatter when merging; use
merged_from field for provenance
- In dry-run mode (the default), CONSOLIDATE, SYNTHESIZE, and GRADUATE describe proposed changes only — no filesystem writes or git operations. The wrapper script sets
DREAM_DRY_RUN_MODE=yes which is substituted into the prompt at runtime.
- GRADUATE commits on a feature branch (
dream/graduate-*), never on main — user reviews and merges
- Maximum 3 graduations per cycle — only entries with confidence >= 0.9 and 3+ observations
Testing
# Dry run (read-only, no filesystem changes — dry-run is the default)
./scripts/auto-dream-cron.sh
# Full run (execute consolidation)
./scripts/auto-dream-cron.sh --execute
# Check output
cat ~/.claude/state/last-dream.md
# Check graduation candidates (what dream would graduate)
python3 -c "
import sys; sys.path.insert(0, 'hooks/lib')
from learning_db_v2 import query_graduation_candidates
import json
candidates = query_graduation_candidates()
print(json.dumps(candidates, indent=2))
"
# Check if a graduation branch exists
git branch --list 'dream/graduate-*'
# Verify cron registration
python3 ~/.claude/scripts/crontab-manager.py list
Cost estimate
$0.09 per nightly run with 50 memory files (20-30K input tokens at Sonnet pricing). ~$33/year for automated overnight operation. Budget capped at $3.00/run via wrapper script.
Cron setup
Use crontab-manager.py (not raw crontab -e) to install. The wrapper script handles PATH, lockfile, logging, budget cap, and dry-run/execute toggle.
# Preview the cron entry
python3 ~/.claude/scripts/crontab-manager.py add \
--tag "auto-dream" \
--schedule "7 2 * * *" \
--command "/home/feedgen/vexjoy-agent/scripts/auto-dream-cron.sh --execute >> /home/feedgen/vexjoy-agent/cron-logs/auto-dream/cron.log 2>&1" \
--dry-run
# Install (after dry-run testing passes)
python3 ~/.claude/scripts/crontab-manager.py add \
--tag "auto-dream" \
--schedule "7 2 * * *" \
--command "/home/feedgen/vexjoy-agent/scripts/auto-dream-cron.sh --execute >> /home/feedgen/vexjoy-agent/cron-logs/auto-dream/cron.log 2>&1"
# Verify
python3 ~/.claude/scripts/crontab-manager.py verify --tag auto-dream
Note: schedule uses 2:07 AM (off-minute) per cron best practice — avoids load spikes from jobs firing at :00.
Wrapper script details
scripts/auto-dream-cron.sh follows the established headless cron pattern (see skills/content/reddit-moderate/scripts/reddit-automod-cron.sh):
flock lockfile prevents concurrent runs
--permission-mode auto (never --dangerously-skip-permissions)
--max-budget-usd 3.00 caps spend per run
--no-session-persistence for clean headless operation
envsubst templates dream-prompt.md with project-specific paths at runtime
tee to timestamped per-run log file
- Dry-run by default,
--execute for live runs
- Exit code propagation via
PIPESTATUS[0]
Reference Loading
Load these references when the task matches the signal:
| Signal / Task |
Reference File |
| Debugging failed cron run, silent failure, empty log, wrong exit code |
references/headless-cron-patterns.md |
Setting up or modifying wrapper script (flock, --permission-mode, envsubst, PIPESTATUS) |
references/headless-cron-patterns.md |
Budget cap, --max-budget-usd, unattended Claude invocation |
references/headless-cron-patterns.md |
| Writing, updating, or archiving memory files |
references/memory-file-operations.md |
Updating MEMORY.md index, atomic write, .tmp rename |
references/memory-file-operations.md |
| Staleness detection, duplicate merging, conflict flagging |
references/memory-file-operations.md |
YAML frontmatter structure, merged_from, memory file format |
references/memory-file-operations.md |
| Testing the dream cycle safely, dry-run validation, output file verification |
references/dream-cycle-testing.md |
| Inspecting graduation candidates, snapshot testing, PIPESTATUS in test wrappers |
references/dream-cycle-testing.md |
| Reading and interpreting cron run logs, detecting silent failures |
references/logging-patterns.md |
| Log rotation, log directory structure, phase completion markers in logs |
references/logging-patterns.md |
last-dream.md stale, missing injection payload, cron log empty |
references/logging-patterns.md |
| Concurrent dream runs, lockfile already held, duplicate cron invocations |
references/concurrency.md |
MEMORY.md.tmp left behind, partial write recovery, atomic rename failure |
references/concurrency.md |
database is locked, SQLite WAL mode, busy_timeout, concurrent DB access |
references/concurrency.md |
local changes would be overwritten, git stash before GRADUATE branch switch |
references/concurrency.md |
1---2name: auto-dream3description: Background memory consolidation and learning graduation — overnight knowledge lifecycle.4---5
6Background memory consolidation cycle. Scans memory files, finds stale/duplicate/conflicting entries, consolidates, synthesizes cross-session insights, builds an injection-ready payload for next session start, and writes a dated dream report.
7
8## When to invoke
9
10- User says "run dream", "consolidate memories", "clean up memories", "memory maintenance", "deduplicate memories"
11- Cron job at 2 AM nightly via wrapper script: `scripts/auto-dream-cron.sh --execute`
12- Manual trigger for testing: `./scripts/auto-dream-cron.sh` (dry-run by default)
13
14## Reference Loading Table
15
16| Signal | Load These Files | Why |
17|---|---|---|
18| Debugging failed cron run, silent failure, empty log, wrong exit code | `headless-cron-patterns.md` | Routes to the matching deep reference |
19| Setting up or modifying wrapper script (`flock`, `--permission-mode`, `envsubst`, `PIPESTATUS`) | `headless-cron-patterns.md` | Routes to the matching deep reference |
20| Budget cap, `--max-budget-usd`, unattended Claude invocation | `headless-cron-patterns.md` | Routes to the matching deep reference |
21| Writing, updating, or archiving memory files | `memory-file-operations.md` | Routes to the matching deep reference |
22| Updating `MEMORY.md` index, atomic write, `.tmp` rename | `memory-file-operations.md` | Routes to the matching deep reference |
23| Staleness detection, duplicate merging, conflict flagging | `memory-file-operations.md` | Routes to the matching deep reference |
24| YAML frontmatter structure, `merged_from`, memory file format | `memory-file-operations.md` | Routes to the matching deep reference |
25| Testing the dream cycle safely, dry-run validation, output file verification | `dream-cycle-testing.md` | Routes to the matching deep reference |
26| Inspecting graduation candidates, snapshot testing, PIPESTATUS in test wrappers | `dream-cycle-testing.md` | Routes to the matching deep reference |
27| Reading and interpreting cron run logs, detecting silent failures | `logging-patterns.md` | Routes to the matching deep reference |
28| Log rotation, log directory structure, phase completion markers in logs | `logging-patterns.md` | Routes to the matching deep reference |
29| `last-dream.md` stale, missing injection payload, cron log empty | `logging-patterns.md` | Routes to the matching deep reference |
30| Concurrent dream runs, lockfile already held, duplicate cron invocations | `concurrency.md` | Routes to the matching deep reference |
31| `MEMORY.md.tmp` left behind, partial write recovery, atomic rename failure | `concurrency.md` | Routes to the matching deep reference |
32| `database is locked`, SQLite WAL mode, `busy_timeout`, concurrent DB access | `concurrency.md` | Routes to the matching deep reference |
33| `local changes would be overwritten`, git stash before GRADUATE branch switch | `concurrency.md` | Routes to the matching deep reference |
34
35## Instructions
36
37When invoked interactively (not via cron), read `skills/meta/auto-dream/dream-prompt.md` and execute its phases directly. The prompt is self-contained — it describes the full seven-phase cycle including safety constraints, file paths, and output formats.
38
39For cron invocation: the dream prompt is passed directly to `claude -p` and runs as a standalone headless session with no CLAUDE.md, no hooks, no project context. All instructions are embedded in the prompt.
40
41## Phases
42
431. **SCAN** — Read all memory files, query learning.db sessions (last 7 days), read recent git log. Write scan document to `~/.claude/state/dream-scan-{date}.md`.
442. **ANALYZE** — Identify stale, duplicate, conflicting memories and cross-session patterns. Write analysis to `~/.claude/state/dream-analysis-{date}.md`.
453. **CONSOLIDATE** — Apply consolidation actions (max 5 changes). Archive stale/merged files, update MEMORY.md atomically.
464. **SYNTHESIZE** — Create insight memories from cross-session patterns (max 2 new memories per cycle).
475. **GRADUATE** — Promote mature learning DB entries (confidence >= 0.9, 3+ observations) into agent/skill files as permanent failure modes. Commits on `dream/graduate-YYYY-MM-DD` branch for human review. Max 3 per cycle. (ADR-159)
486. **SELECT** — Build injection-ready payload for session start. Write to `~/.claude/state/dream-injection-{project-hash}.md`.
497. **REPORT** — Write dream summary to `~/.claude/state/last-dream.md`.
50
51## Safety constraints (always enforced)
52
53- Never delete files — archive to `memory/archive/`, never `rm`
54- Write the REPORT before executing any CONSOLIDATE filesystem operations
55- Maximum 5 memory changes per cycle — excess items deferred to next cycle
56- Flag conflicts for human review, never auto-resolve
57- Preserve YAML frontmatter when merging; use `merged_from` field for provenance
58- In dry-run mode (the default), CONSOLIDATE, SYNTHESIZE, and GRADUATE describe proposed changes only — no filesystem writes or git operations. The wrapper script sets `DREAM_DRY_RUN_MODE=yes` which is substituted into the prompt at runtime.
59- GRADUATE commits on a feature branch (`dream/graduate-*`), never on main — user reviews and merges
60- Maximum 3 graduations per cycle — only entries with confidence >= 0.9 and 3+ observations
61
62## Testing
63
64```bash
65# Dry run (read-only, no filesystem changes — dry-run is the default)
66./scripts/auto-dream-cron.sh
67
68# Full run (execute consolidation)
69./scripts/auto-dream-cron.sh --execute
70
71# Check output
72cat ~/.claude/state/last-dream.md
73
74# Check graduation candidates (what dream would graduate)
75python3 -c "
76import sys; sys.path.insert(0, 'hooks/lib')
77from learning_db_v2 import query_graduation_candidates
78import json
79candidates = query_graduation_candidates()
80print(json.dumps(candidates, indent=2))
81"
82
83# Check if a graduation branch exists
84git branch --list 'dream/graduate-*'
85
86# Verify cron registration
87python3 ~/.claude/scripts/crontab-manager.py list
88```
89
90## Cost estimate
91
92~$0.09 per nightly run with 50 memory files (~20-30K input tokens at Sonnet pricing). ~$33/year for automated overnight operation. Budget capped at $3.00/run via wrapper script.
93
94## Cron setup
95
96Use `crontab-manager.py` (not raw `crontab -e`) to install. The wrapper script handles PATH, lockfile, logging, budget cap, and dry-run/execute toggle.
97
98```bash
99# Preview the cron entry
100python3 ~/.claude/scripts/crontab-manager.py add \
101 --tag "auto-dream" \
102 --schedule "7 2 * * *" \
103 --command "/home/feedgen/vexjoy-agent/scripts/auto-dream-cron.sh --execute >> /home/feedgen/vexjoy-agent/cron-logs/auto-dream/cron.log 2>&1" \
104 --dry-run
105
106# Install (after dry-run testing passes)
107python3 ~/.claude/scripts/crontab-manager.py add \
108 --tag "auto-dream" \
109 --schedule "7 2 * * *" \
110 --command "/home/feedgen/vexjoy-agent/scripts/auto-dream-cron.sh --execute >> /home/feedgen/vexjoy-agent/cron-logs/auto-dream/cron.log 2>&1"
111
112# Verify
113python3 ~/.claude/scripts/crontab-manager.py verify --tag auto-dream
114```
115
116Note: schedule uses 2:07 AM (off-minute) per cron best practice — avoids load spikes from jobs firing at :00.
117
118## Wrapper script details
119
120`scripts/auto-dream-cron.sh` follows the established headless cron pattern (see `skills/content/reddit-moderate/scripts/reddit-automod-cron.sh`):
121- `flock` lockfile prevents concurrent runs
122- `--permission-mode auto` (never `--dangerously-skip-permissions`)
123- `--max-budget-usd 3.00` caps spend per run
124- `--no-session-persistence` for clean headless operation
125- `envsubst` templates `dream-prompt.md` with project-specific paths at runtime
126- `tee` to timestamped per-run log file
127- Dry-run by default, `--execute` for live runs
128- Exit code propagation via `PIPESTATUS[0]`
129
130## Reference Loading
131
132Load these references when the task matches the signal:
133
134| Signal / Task | Reference File |
135|---------------|----------------|
136| Debugging failed cron run, silent failure, empty log, wrong exit code | `references/headless-cron-patterns.md` |
137| Setting up or modifying wrapper script (`flock`, `--permission-mode`, `envsubst`, `PIPESTATUS`) | `references/headless-cron-patterns.md` |
138| Budget cap, `--max-budget-usd`, unattended Claude invocation | `references/headless-cron-patterns.md` |
139| Writing, updating, or archiving memory files | `references/memory-file-operations.md` |
140| Updating `MEMORY.md` index, atomic write, `.tmp` rename | `references/memory-file-operations.md` |
141| Staleness detection, duplicate merging, conflict flagging | `references/memory-file-operations.md` |
142| YAML frontmatter structure, `merged_from`, memory file format | `references/memory-file-operations.md` |
143| Testing the dream cycle safely, dry-run validation, output file verification | `references/dream-cycle-testing.md` |
144| Inspecting graduation candidates, snapshot testing, PIPESTATUS in test wrappers | `references/dream-cycle-testing.md` |
145| Reading and interpreting cron run logs, detecting silent failures | `references/logging-patterns.md` |
146| Log rotation, log directory structure, phase completion markers in logs | `references/logging-patterns.md` |
147| `last-dream.md` stale, missing injection payload, cron log empty | `references/logging-patterns.md` |
148| Concurrent dream runs, lockfile already held, duplicate cron invocations | `references/concurrency.md` |
149| `MEMORY.md.tmp` left behind, partial write recovery, atomic rename failure | `references/concurrency.md` |
150| `database is locked`, SQLite WAL mode, `busy_timeout`, concurrent DB access | `references/concurrency.md` |
151| `local changes would be overwritten`, git stash before GRADUATE branch switch | `references/concurrency.md` |