Smoke Test Skillpack
Walk the check list below after any container restart. There is no smoke-test binary — every check is a real
memraincommand you run and read.
Contract
This skill guarantees:
- 6 core checks verify Memrain server health after restart
- Each check is a real command, not a wrapper script
- Known failures are auto-fixed before reporting
- Results are reported as a single summary line plus per-check status
- The reported failure count = checks still broken after auto-fix (0 = all pass)
The Check List
| # | Check | Command | Auto-Fix |
|---|---|---|---|
| 1 | Server answers + config sane | memrain status |
Restart the service |
| 2 | Database, sources, brain health | memrain doctor |
Per-finding (doctor names the cause) |
| 3 | Background cycle ran recently | memrain status (cycle timestamp) |
Kick a manual memrain cycle |
| 4 | Secrets + Bedrock reachable | memrain doctor (auth findings) |
Re-run the fetch-secrets step, restart |
| 5 | Read path returns hits | memrain call search --args '{"q":"<known topic>"}' |
Re-index, then memrain embed |
| 6 | Write path round-trips | page_put → page_get (see below) |
— (escalate; a broken write path is not self-healing) |
Checks 1-4 are read-only. Check 5 reads. Check 6 writes one throwaway page and reads it back.
Usage
From a shell on the host
memrain status
memrain doctor
memrain call search --args '{"q":"<a topic you know is indexed>"}'
Write-path round-trip (check 6)
memrain call page_put --args '{"slug":"reports/smoke/_probe","markdown_body":"# probe\nsmoke check"}'
memrain call page_get --args '{"slug":"reports/smoke/_probe"}'
The page_get must return the content the page_put just wrote. Reuse the
same slug every run — the probe page is overwritten, not accumulated.
From an agent without shell access
The same checks run over MCP: get_status_snapshot (1, 3), run_doctor (2, 4),
search (5), page_put + page_get (6). Read-only checks are identical; the
auto-fixes that need a service restart aren't available over MCP — report them
instead.
From host bootstrap
Bootstrap can't run judgment. Have it run the two commands that exit non-zero on real trouble and log them for a human or agent to read:
memrain status >> /tmp/bootstrap.log 2>&1
memrain doctor >> /tmp/bootstrap.log 2>&1
Adding a Check
There's no script to edit — a new check is a new row in the table above plus the command that proves it. Follow this pattern:
N. [What it proves]
Probe: memrain call <tool> --args '<json>' # or a memrain subcommand
Pass: [the specific field/shape that means healthy]
Auto-fix: [the command] → re-run the probe → still bad? report it
Prefer memrain call <tool> over raw HTTP: it goes through the same dispatch
the MCP clients use, so a green check means the surface agents actually use
is green.
Design rules:
- Probe first — never fix without confirming broken
- Re-probe after fix — verify the fix worked
- Bound every command — wrap anything that could hang in
timeout N - One line per check — status, check name, and the detail if it failed
- Idempotent fixes — safe to run repeatedly
- Skip gracefully — a missing prerequisite is a skip, not a failure
Configuration
The checks carry no knobs of their own — they read whatever the server is
already configured with. memrain status prints the resolved configuration
(sources, model tiers, region, cycle state); memrain doctor names the config
that's wrong when a check fails. If a check needs a value, get it from those
two commands rather than assuming an env var.
Known Issues & Their Auto-Fixes
Secrets not yet fetched after restart
- Symptom: server boots but
/healthreports degraded; Bedrock and DB calls fail with auth errors - Cause: the container restarted before the secrets-fetch step wrote the secrets files, so the process started with an empty env
- Auto-fix: re-run the fetch-secrets step, then restart the service
- Persistence: does NOT survive container restart (the race can recur on every boot)
- This is why smoke tests must run on every restart
Worker DB Auth Failure
- Symptom: background jobs can't connect to the DB
- Cause: the database URL isn't propagated to the worker subprocess
- Auto-fix: restart the service so the worker inherits the fetched env;
memrain doctorconfirms the job surface recovered
Anti-Patterns
- ❌ Running smoke checks on every chat turn. Once per container restart (or on user request) is plenty. The checks are cheap but they're not free.
- ❌ Running a probe without
timeout Naround anything that could hang. One hung probe stalls every check after it. - ❌ Auto-fixing without confirming the check is actually broken first.
The
probe → fail-detected → fix → re-probeloop is the contract; fixes that skip the re-probe can report success on a still-broken state. - ❌ Treating a skip as a failure. Missing prerequisites (no note source mounted, no worker configured) are skips. The reported failure count is real failures only.
- ❌ Inventing a command to make a check look green. Every check in the list
above is a command that exists; if you need a new probe, use
memrain call <tool>against a real tool rather than a script that isn't there. - ❌ Leaving the check-6 probe page behind under a fresh slug each run. Reuse one slug so the brain doesn't accumulate smoke debris.
Output Format
Report one line per check (✅ / ❌ / 🔧 fixed / ⏭️ skipped) naming the check and,
on failure, the command output that proved it broken. Close with a single
summary line: Results: N/M passed, F auto-fixed, S skipped. When any check
is still red after its auto-fix, say which ones and what the fix attempt
returned — a smoke report that hides the failing command is worse than no
report.