Testing Assistant Conversations

Testing assistant conversations, agents, and skills

This guide matches patterns in packages/core/assistant-toolkit and related packages (assistant, plugin-markdown, plugin-assistant). For regenerating model fixtures only, prefer the focused skill regenerate-model-fixture.

AssistantTestLayer

Import from @dxos/assistant/testing.

AssistantTestLayer composes:

• AI — TestAiService (memoized by default; see below), default model ai.claude.model.claude-opus-4-6.
• Tool execution — ToolExecutionServices and OpaqueToolkit.providerLayer.
• Skill registry — Skill.RegistryService seeded with optional skills.
• Operations — operationHandlers passed to OperationHandlerSet.provide(...); ProcessManager wires Operation.Service for tool execution (see AssistantTestLayer in packages/core/assistant/src/testing/layer.ts).
• ECHO test DB — TestDatabaseLayer with types you register.
• Credentials — CredentialsService.configuredLayer(credentials) (often [] in tests).
• Tracing — noop | console | pretty.

Use AssistantTestLayerWithTriggers when the scenario uses scheduled triggers (manual time control, in-memory trigger state). Example: packages/core/assistant-toolkit/src/skills/project/skill.test.ts.

Important options

Implementation reference: packages/core/assistant/src/testing/layer.ts.

Model memoization and DX_UPDATE_MODEL_FIXTURES

AssistantTestLayer includes memoization internally — you do not need to set up LanguageModelFixture yourself. The layer wraps the AI service with LanguageModelFixture.layerTest automatically (unless disableLlmMemoization: true).

Default test AI goes through LanguageModelFixture.layerTest, which:

• Writes/reads hash-addressed fixtures under .store/conversations/<suite>/<hash>.json (suite path from TestContextService).
• Without DX_UPDATE_MODEL_FIXTURES: replays only; missing matching prompt → error telling you to regenerate.
• With DX_UPDATE_MODEL_FIXTURES=1 (or true): calls the real model when no match exists and updates the JSON.

CI stays deterministic because it uses committed fixtures, not live LLM calls.

Requirements for regeneration

1. Credentials — API keys must be in the environment. In this repo, load 1Password-injected env from the workspace root:

   • fish: eval (pnpm -ws 1p-credentials)
   • bash/zsh: eval "$(pnpm -ws 1p-credentials)"

   The script is the 1p-credentials package script (runs op inject over inlined op://CI/… references; the full env template lives in the dxos/edge repo as .env.tpl).

2. Run tests with generation:

   DX_UPDATE_MODEL_FIXTURES=1 moon run assistant-toolkit:test
   

   Or all memoized-LLM packages: DX_UPDATE_MODEL_FIXTURES=1 moon run '#model-fixture:test'.

3. Commit updated .store/conversations/** files.

Packages that participate are tagged model-fixture in their moon.yml (e.g. assistant-toolkit, assistant, ai, plugin-markdown, plugin-assistant).

Timeouts

LLM conversation tests should use a longer timeout to account for generation. Pattern: { timeout: 60_000 } or LanguageModelFixture.isUpdateEnabled() ? 240_000 : 30_000. Note that LanguageModelFixture is only needed as an import for the timeout helper — the layer already handles memoization internally.

TestHelpers.provideTestContext

Effects that use memoization must end with TestHelpers.provideTestContext (from @dxos/effect/testing) so the memo layer knows the current test file path. Typical pipe:

Effect.fnUntraced(..., Effect.provide(TestLayer), TestHelpers.provideTestContext).

Real LLM calls and DX_ANTHROPIC_API_KEY

The default aiServicePreset: 'direct' calls the Anthropic API directly. Set DX_ANTHROPIC_API_KEY (via pnpm -ws 1p-credentials or export DX_ANTHROPIC_API_KEY=sk-ant-...) when regenerating the memoized cache with DX_UPDATE_MODEL_FIXTURES=1. Use DX_ANTHROPIC_API_KEY, not ANTHROPIC_API_KEY (the latter breaks Claude Code). Normal cached runs need no key. Works for both direct operation invocations and full conversation tests. Example: packages/core/assistant-toolkit/src/skills/skill-manager/skill.test.ts.

General test structure

Vitest + Effect

Use @effect/vitest (describe, it.effect, it.scoped) and Effect.fnUntraced for generator bodies.

Determinism

Many tests call EntityId.dangerouslyDisableRandomness() at module scope for stable IDs. The PRNG is shared across all tests in the same file — memos and fixtures that embed object IDs only match when tests run in file order. When regenerating memoized LLM cache, never use vitest -t for a single test; regenerate the whole test file (see regenerate-model-fixture skill).

Database and invocation flow

1. yield* Database.add(...) / Obj.make(...) for fixtures.
2. yield* Database.flush() before invoking functions or conversations that read persisted state.
3. Call Operation.invoke(Operation, input) for direct operation tests, or AiSessionService.run, new AiSession, AiRequest, etc., depending on the layer under test.

Registering skills in tests

Two common patterns:

1. Registry at layer build — pass skills: [SomeSkill.make(), ...] into AssistantTestLayer when services read from the registry.

2. Runtime bind — addSkills from packages/core/assistant-toolkit/src/skills/testing.ts loads definition make() objects into the DB and calls AiContextService.bindContext({ skills: [...] }). Used with AiSessionService.layerNewFeed().pipe(Layer.provideMerge(TestLayer)) in memory skill tests.

You still pass the skill’s operations (handler set) into AssistantTestLayer({ operationHandlers: ... }) so tools actually execute.

Types list

Include every ECHO type instances may have: skill metadata types, domain objects (Message, Person, plugin documents), Skill.Skill, Trigger.Trigger, queues, etc. If in doubt, mirror imports from a similar test in the same skill folder.

Quick checklist

• AssistantTestLayer (or WithTriggers) with correct operationHandlers and types.
• Effect.provide(TestLayer) + TestHelpers.provideTestContext for memoized LLM tests.
• New/changed prompts → regenerate with DX_UPDATE_MODEL_FIXTURES=1 + 1p-credentials, commit .store/conversations/**.
• Package has model-fixture tag if tests use memoization (for CI grouping).