Writing Live E2e Tests

Writing Live E2E Tests

Overview

Live E2E tests run against a real Slack workspace via Socket Mode. Each test is a standalone run-*.ts file in src/e2e/live/ that exports a LiveE2EScenario object. The CLI auto-discovers and runs them.

When the user asks to add a live E2E test, implement the scenario and run the local validation commands. When they ask to "跑下" or run it and .env.e2e exists, run the real live scenario too.

Skeleton

Every scenario file follows this structure:

import './load-e2e-env.js';
import { randomUUID } from 'node:crypto';
import fs from 'node:fs/promises';
import path from 'node:path';
import { createApplication } from '~/application.js';
import { env } from '~/env/server.js';
import type { LiveE2EScenario } from './scenario.js';
import { runDirectly } from './scenario.js';
import { SlackApiClient } from './slack-api-client.js';

interface MyResult {
  botUserId: string;
  channelId: string;
  failureMessage?: string;
  matched: {
    /* booleans for each assertion */
  };
  passed: boolean;
  rootMessageTs?: string;
  runId: string;
}

async function main(): Promise<void> {
  // 1. Guard env
  if (!env.SLACK_E2E_ENABLED) throw new Error('...');
  if (!env.SLACK_E2E_CHANNEL_ID || !env.SLACK_E2E_TRIGGER_USER_TOKEN) throw new Error('...');

  // 2. Setup
  const runId = randomUUID();
  const triggerClient = new SlackApiClient(env.SLACK_E2E_TRIGGER_USER_TOKEN);
  const botClient = new SlackApiClient(env.SLACK_BOT_TOKEN);
  const botIdentity = await botClient.authTest();

  // 3. Init result object with all matched: false
  const result: MyResult = {
    /* ... */
  };

  // 4. Start application
  const application = createApplication();
  let caughtError: unknown;

  try {
    await application.start();
    await delay(3_000);

    // 5. Post trigger message with runId marker
    // 6. Poll with deadline loop
    // 7. Assert via assertResult()
    // 8. Set result.passed = true AFTER assertion passes

    await writeResult(result);
    assertResult(result);
    result.passed = true;
    await writeResult(result);
  } catch (error) {
    result.failureMessage = error instanceof Error ? error.message : String(error);
    caughtError = error;
  } finally {
    await writeResult(result).catch(() => {});
    await application.stop().catch(() => {});
  }
  if (caughtError) throw caughtError;
}

// ALWAYS use env.SLACK_E2E_RESULT_PATH with .replace()
async function writeResult(result: MyResult): Promise<void> {
  const resultPath = env.SLACK_E2E_RESULT_PATH.replace(/result\.json$/, 'my-test-result.json');
  const absolutePath = path.resolve(process.cwd(), resultPath);
  await fs.mkdir(path.dirname(absolutePath), { recursive: true });
  await fs.writeFile(absolutePath, `${JSON.stringify(result, null, 2)}\n`, 'utf8');
}

function assertResult(result: MyResult): void {
  const failures: string[] = [];
  // Push descriptive failure strings
  if (failures.length > 0) throw new Error(`E2E failed: ${failures.join('; ')}`);
}

function delay(ms: number): Promise<void> {
  return new Promise((resolve) => {
    setTimeout(resolve, ms);
  });
}

export const scenario: LiveE2EScenario = {
  id: 'kebab-case-id',
  title: 'Human Readable Title',
  description: 'One sentence describing what is verified.',
  keywords: ['searchable', 'terms'],
  run: main,
};

runDirectly(scenario);

Key Rules

Polling Pattern

const deadline = Date.now() + env.SLACK_E2E_TIMEOUT_MS;
while (Date.now() < deadline) {
  const replies = await botClient.conversationReplies({
    channel: env.SLACK_E2E_CHANNEL_ID,
    inclusive: true,
    limit: 50,
    ts: rootMessage.ts,
  });

  for (const message of replies.messages ?? []) {
    if (!message.ts || message.ts === rootMessage.ts) continue;
    if (message.user === botIdentity.user_id || message.bot_id) {
      const text = typeof message.text === 'string' ? message.text : '';
      if (text.includes(`MY_MARKER ${runId}`)) {
        result.matched.myCondition = true;
      }
    }
  }

  if (result.matched.myCondition) break;
  await delay(2_500);
}

Advanced Patterns

Anchor-then-prompt (file uploads, multi-step)

Post a non-mention anchor first, upload files or set up state, then post the mention prompt in-thread.

Multi-phase tests

Phase 1 saves state → restart application → Phase 2 verifies persistence. Each phase gets its own root message and polling loop.

Reaction lifecycle

Poll getReactions() for emoji presence/absence across phases (ack added → removed → done added).

Database validation

Use better-sqlite3 to query SQLite directly after bot processes, verify persistence of memory/session records.

Important schema details:

• memories has repo_id, not scope.
• Global memories are repo_id IS NULL.
• Workspace memories use repo_id = ?.
• Resolve DB path with path.resolve(process.cwd(), env.SESSION_DB_PATH).

Status probe

FileSlackStatusProbe writes NDJSON records. Read the probe file to verify tool progress events.

Use resetSlackStatusProbeFile(env.SLACK_E2E_STATUS_PROBE_PATH) before posting the trigger. After polling, filter records by record.threadTs === rootMessage.ts.

For Slack-visible rendering regressions, collect:

• status records: record.status and record.loadingMessages
• progress records: record.text

Assert both the positive user-facing text and absence of raw internals. Example: for Codex memory file writes, assert Saving memory... appears and /bin/zsh -lc, .kagura/runtime, and memory-ops.jsonl do not.

Validation Commands

After editing or adding a live scenario, run these from the repository root:

pnpm --filter kagura exec tsc --noEmit
pnpm exec prettier --check apps/kagura/src/e2e/live/run-your-scenario.ts

Run targeted unit tests too when the scenario covers code with existing tests, for example:

pnpm --filter kagura exec vitest run tests/codex-cli-adapter.test.ts

Check scenario discovery with required dummy env if the local shell has no live secrets loaded:

SLACK_BOT_TOKEN=xoxb-test SLACK_APP_TOKEN=xapp-test SLACK_SIGNING_SECRET=secret REPO_ROOT_DIR=/Users/innei/git/innei-repo pnpm --filter kagura exec tsx src/e2e/live/cli.ts --list your-search-term

Running Real Live E2E

Prefer running from the repository root. Because pnpm --filter kagura exec executes in the package directory, load-e2e-env.ts may not find the root .env.e2e by relative path. If .env.e2e exists at the repository root, source it explicitly:

set -a
source .env.e2e
set +a
pnpm --filter kagura exec tsx src/e2e/live/cli.ts scenario-id

If the CLI reports dotenv injected 0 variables and env validation says SLACK_BOT_TOKEN, SLACK_APP_TOKEN, SLACK_SIGNING_SECRET, or REPO_ROOT_DIR are missing, rerun with the explicit source .env.e2e command above.

Expected noisy but usually non-fatal live logs:

• Slack manifest token rotation may fail; slash command sync is skipped, but app mention E2E can still run.
• Socket Mode may log WebSocket ping/pong warnings.
• After application.stop(), Bolt may log "client has no active connection" for late events. If the scenario summary says PASS, treat this as non-blocking unless the task is specifically about shutdown behavior.

Common Mistakes