User Test Memory

/user-test-memory - Multi-Model Memory User Testing

Spawn independent subagents at different capability levels to stress-test the Munin memory MCP tools. Each agent gets minimal briefing and must figure things out from tool descriptions and exploration. This surfaces UX issues, broken flows, confusing behavior, and missing capabilities.

Default mode: 3 Claude agents (Opus/Sonnet/Haiku). Extended mode adds Codex (via codex exec) and Antigravity (via agy --print) — see Cross-Ecosystem Extension below.

Usage

• /user-test-memory - Full exploratory test across all tools (3 Claude agents)
• /user-test-memory retrieval - Focus testing on search/query
• /user-test-memory onboarding - Focus on first-time orientation experience
• /user-test-memory writing - Focus on write/log/update flows
• /user-test-memory full - 9-runner cross-ecosystem test (Claude + Codex + Antigravity)

How It Works

Step 1: Generate Test Session ID

test-<YYYYMMDD-HHMMSS>

Each agent gets a unique test namespace: testing/<model>-<session-id> (e.g., testing/opus-test-20260408-120000).

Step 2: Build the Agent Briefing

Each agent gets the SAME briefing — a deliberately minimal context that simulates a new user. Do NOT include CLAUDE.md conventions, namespace patterns, tag vocabulary, or usage guides. The agents should discover these from the tools themselves.

Briefing template (fill in {test_namespace}, {session_id}, and {focus_area} if provided):

You are a QA tester evaluating a memory system called Munin. You have access to
MCP tools prefixed with "mcp__munin-memory__". Your job is to explore these tools
as a first-time user and report what works well, what's confusing, and what's broken.

Your test namespace is "{test_namespace}" — use this for any writes so you don't
pollute real data. You may READ from any namespace to understand the system.

{focus_area_instruction}

## Your Mission

Work through these test phases. Spend roughly equal effort on each.

### Phase 1: Orient (understand the system)
- Call memory_orient. What do you learn? Is it helpful? Overwhelming? Missing things?
- Try memory_list to browse. Can you understand the namespace structure?
- Pick 2-3 interesting namespaces and read their contents.

### Phase 2: Write (create some test data)
- Write a state entry to your test namespace
- Log a couple of entries
- Try updating an existing entry
- Try writing with tags
- Try anything that seems like it should work — does it?

### Phase 3: Retrieve (find things)
- Use memory_query to search for something you just wrote
- Search for something that exists in the system already
- Try different search modes if available
- Try filtering by tags, time ranges, namespaces

### Phase 4: Advanced tools (if they exist)
- Explore any tools you haven't tried yet
- Try edge cases: empty queries, very long content, special characters
- Look for tools that seem powerful but unclear how to use

### Phase 5: Stress test
- Try something you think SHOULD work but aren't sure about
- Try to break something (within your test namespace)
- Look for inconsistencies between tool descriptions and actual behavior

## Report Format

End your exploration with a structured report. Be specific — cite exact tool names,
error messages, and surprising behaviors. Rate each category:
  S = Superb (delightful, better than expected)
  A = Good (works well, minor polish needed)
  B = Acceptable (works but has rough edges)
  C = Poor (confusing, hard to use, or unreliable)
  F = Broken (doesn't work, produces wrong results, or crashes)

```markdown
# Munin Memory User Test Report — {model_name}

## Scores
- Onboarding/Orientation: [S/A/B/C/F]
- Writing & Updating: [S/A/B/C/F]
- Reading & Retrieval: [S/A/B/C/F]
- Tool Discoverability: [S/A/B/C/F]
- Error Messages & Feedback: [S/A/B/C/F]
- Overall Coherence: [S/A/B/C/F]

## What Works Great
<specific things that were impressive or well-designed>

## What's Confusing
<things that weren't intuitive, unclear tool descriptions, surprising behavior>

## What's Broken
<bugs, crashes, wrong results, inconsistencies>

## What's Missing
<capabilities you expected but didn't find, workflows that felt incomplete>

## Verbatim Tool Friction Log
<chronological log of moments where you got stuck, had to retry, or were surprised>

## Top 3 Recommendations
1. <highest impact improvement>
2. <second>
3. <third>

### Step 3: Determine Focus Area Instruction

If a focus area was provided, add this to the briefing:

| Focus | Instruction |
|-------|-------------|
| `retrieval` | "Spend 60% of your effort on Phase 3 (retrieval). Test query reformulations, semantic vs lexical search, temporal filters, tag filters, and result quality." |
| `onboarding` | "Spend 60% of your effort on Phase 1 (orient). Evaluate whether a brand-new agent can understand the system from orient alone, without any documentation." |
| `writing` | "Spend 60% of your effort on Phase 2 (writing). Test edge cases in writes, updates, patches, tag handling, validation errors, and content limits." |
| (none) | "Distribute your effort roughly equally across all phases." |

### Step 4: Launch All Three Agents in Parallel

Spawn three Agent tool calls in a SINGLE message (parallel execution):

1. **Opus** (`model: "opus"`) — the deep thinker. Will find subtle design issues.
2. **Sonnet** (`model: "sonnet"`) — the practical middle. Represents the typical AI user.
3. **Haiku** (`model: "haiku"`) — the constrained model. If Haiku can't figure it out, the UX needs work.

Each agent gets the same briefing with its own `{test_namespace}`.

**Claude-specific note:** MCP tools are deferred — prepend the briefing with:
> "First call ToolSearch with query `select:mcp__munin-memory__memory_orient,...` (list the key tools) to load their schemas, then proceed."

Agent description format: `"Memory user test ({model})"`.

### Step 5: Collect and Synthesize Reports

Once all three agents return:

1. **Extract each report** from the agent results
2. **Build a synthesis document** that cross-references findings:

```markdown
# Munin Memory User Testing — {date}

## Test Configuration
- Session: {session_id}
- Focus: {focus_area or "full exploratory"}
- Models: Opus, Sonnet, Haiku

## Score Matrix

| Category | Opus | Sonnet | Haiku | Consensus |
|----------|------|--------|-------|-----------|
| Onboarding | | | | |
| Writing | | | | |
| Retrieval | | | | |
| Discoverability | | | | |
| Error Messages | | | | |
| Overall | | | | |

## Consensus Findings (2+ models agree)
<findings reported by at least 2 of the 3 models>

## Unique Findings
### Opus only
<issues only Opus caught — likely subtle or architectural>

### Sonnet only
<issues only Sonnet caught>

### Haiku only
<issues only Haiku caught — likely UX/clarity issues>

## Bugs Found
<concrete bugs with reproduction steps>

## Prioritized Recommendations
1. <highest impact, based on consensus + severity>
2. ...

## Raw Reports
<include each model's full report below for reference>

3. Save the synthesis to docs/user-testing-{session_id}.md in the current repo (if in munin-memory) or print it to the conversation.

Step 6: Offer Cleanup

After presenting results, offer to clean up test namespaces:

memory_delete("testing/opus-{session_id}")
memory_delete("testing/sonnet-{session_id}")
memory_delete("testing/haiku-{session_id}")

Cross-Ecosystem Extension

For a richer signal, run the test across Claude + Codex (OpenAI) + Antigravity (Gemini) in a Workflow. This surfaces ecosystem-specific UX issues that a single-family run can't catch.

Runner families

Model note (verified 2026-07-25, codex-cli 0.145.0): ChatGPT-account Codex auth exposes only gpt-5.6-sol and gpt-5.5. Everything else — gpt-5.6, gpt-5.6-codex, gpt-5.5-codex, gpt-5.6-pro, *-mini, gpt-5.x, o3 — returns 400 … not supported when using Codex with a ChatGPT account. Always re-probe before a run: codex exec --skip-git-repo-check -s read-only -m <slug> "Reply with only: PROBE_OK", and check the output is NOT echoed from the prompt.

Build tiers from reasoning effort, not model count. -c model_reasoning_effort= accepts low / medium / high / xhigh (minimal errors). A 5-runner Codex fleet is e.g. sol@xhigh, sol@high, sol@low, g55@high, g55@low. Effort is the dominant variable: in the 2026-07-25 run the xhigh runner was the only one to find a data-loss delete race and to read consolidation output critically; the low-effort runners still found every consensus issue, so the low tiers are worth keeping.

Codex runners must NOT run inside the repo under test. codex exec auto-loads AGENTS.md/CLAUDE.md from the working directory, which hands the runner the full tool inventory, namespace grammar and conventions — invalidating the discoverability grade. Run from a neutral scratch directory and pass -c project_doc_max_bytes=0. (The global ~/.codex/AGENTS.md still loads; that is acceptable, it carries handshake discipline but not the tool surface.)

Codex write access works headlessly despite approval_mode = "approve" on the munin tools in ~/.codex/config.toml — no --dangerously-bypass-approvals-and-sandbox needed. Use -s workspace-write with cwd = the scratch dir so runners can also write their report file; ask for the report both as a file and as the final message.

Namespace rules (critical)

Namespace grammar: [a-zA-Z0-9][a-zA-Z0-9/_-]* — dots and spaces are invalid. Runner IDs used in sandbox namespace names MUST be sanitised. Use codex-a, codex-b, codex-c — NOT codex-gpt5.5-a (the dot breaks the namespace and blocks the entire write phase with no warning).

Briefing variants

• Claude: standard briefing (MCP tools natively available; prepend ToolSearch hint).
• Codex: same briefing written to /tmp/brief-<id>.txt, piped to codex exec. Codex resolves as principal: owner via the munin bridge in ~/.codex/config.toml.
• Antigravity (read-only): modified briefing that instructs the runner to skip write tools (memory_write, memory_log, memory_update_status, memory_delete) and mark Writing grade as "N/A — could not test headlessly". agy resolves munin tools without a mcp__-prefix (uses generic call_mcp_tool).

De-risk before fan-out

Before launching 9 parallel runners, verify each family can actually call munin:

1. Claude: no de-risk needed (native).
2. Codex: codex exec -m gpt-5.5 "Call memory_status with no args and paste the result." — confirm it returns JSON, not a sandbox/network error.
3. agy: agy --print "List the MCP tools available to you." — confirm munin tools appear. Then agy --print "Call memory_status (no args) and paste the raw result." — confirm a live response.

Synthesis weighting

When synthesising across families, weight cross-family consensus higher than within-family agreement. Two different ecosystems independently reporting the same issue is near-certain signal. One family reporting something the others didn't may still be worth filing, but note the limited confidence.

Cleanup

After the run, delete all testing/* sandbox namespaces. Codex and agy runners connect as principal: owner, so their writes land in prod munin — clean up even if you think they failed.

Cleanup is currently not fully possible (verified 2026-07-25, munin-memory v0.6.1, filed as munin-memory#279): namespace-wide delete returns namespace_delete_disabled unless MUNIN_ALLOW_NAMESPACE_DELETE=true is set server-side, and single-entry delete needs namespace + key, so log entries cannot be deleted at all. Budget for this: a 5-runner run leaves ~16 namespaces and ~80 state entries behind. Either accept the residue, or ask Magnus to flip the flag before the run. Do delete anything a runner wrote outside its sandbox — one runner escaped testing/* to test memory_update_status and created a live projects/<runner-id>/status.

Warn runners explicitly not to work around a sandbox restriction by writing elsewhere. Add it to the briefing, not just the report format.

Key Rules

1. Minimal briefing — The whole point is testing discoverability. Don't teach the agents how to use the system.
2. Parallel launch — All three agents MUST be launched in a single message for parallel execution.
3. Test namespaces — All writes go to testing/* to avoid polluting production data. Sanitise runner IDs — no dots or spaces in namespace names.
4. Read access is fine — Agents can and should read real data to evaluate the system.
5. No coaching — If an agent gets stuck, that's a finding, not a problem to solve.
6. Specific feedback — Vague feedback ("it was fine") is useless. The report format enforces specificity.
7. Cross-model comparison — The synthesis matters more than individual reports. Consensus findings are high-confidence signals.
8. De-risk external runners first — Verify Codex and agy can reach and call munin with a single-tool probe before running the full fan-out.
9. Verify the runners' claims before filing — reports disagree, and confident reports are sometimes wrong. In the 2026-07-25 Codex run two runners gave contradictory accounts of limit-clamping, and one asserted "no warning" where the server does warn. Check each finding against the source (or reproduce it) before it becomes a ticket; cite the confirming file/line in the issue.
10. Concurrent runners contaminate the review inbox — every runner authenticates as principal: owner, so memory_review list shows every other runner's pending proposals. That is itself a finding, but it also means proposal-workflow results are not independent across runners.