Testing

Testing & Verification

Quick Reference

Verification Workflow

After making changes, run the narrowest credible test first, then broader checks.

Narrow Tests by Crate

Multiple crates: cargo test -p runtimed -p notebook-doc.

MCP Live Verification (when nteract-dev available)

For daemon/kernel changes: up rebuild=true → create_notebook → create_cell with 1 + 1 → execute_cell → verify output is 2.

For CRDT/doc changes: create_notebook → create_cell → get_cell (verify source) → set_cell → get_cell (verify update).

For kernel-env changes: up rebuild=true → create_notebook → execute import sys; print(sys.executable) → verify Python path.

Reporting verification

State which checks ran and what they established. Distinguish compilation, narrow tests, and MCP live verification; name skipped or blocked checks and the behavior they leave unverified.

For example, after narrow tests pass but a live check is unavailable:

The daemon tests passed. I couldn't run the MCP check because nteract-dev wasn't available, so live cell execution is still unverified.

Always run cargo xtask lint --fix before committing.

Frontend Unit Tests (Vitest)

Config: vitest.config.ts (jsdom environment, globals enabled).

pnpm test         # Watch mode
pnpm test:run     # Run once

Key locations: src/components/isolated/__tests__/, src/components/outputs/__tests__/, src/components/widgets/__tests__/, apps/notebook/src/lib/__tests__/.

Rust Unit Tests

cargo test                    # All workspace tests
cargo test -p runtimed        # Specific crate
cargo test -- --nocapture     # Show println! output

Hone CLI Tests

Declarative bash-based tests in crates/runt/tests/*.hone.

cargo hone test               # All hone tests
cargo hone test cli.hone      # Specific file

Assertions: ASSERT exit_code == 0, ASSERT stdout contains "text", ASSERT stdout matches /pattern/.

Python Tests

Two venvs: workspace (.venv at root) for dev, and python/runtimed/.venv for isolated pytest.

# Setup test venv
cd python/runtimed && python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
cd ../../crates/runtimed-py && VIRTUAL_ENV=../../python/runtimed/.venv maturin develop

# Run
pytest python/runtimed/tests/test_session_unit.py -v          # Unit (no daemon)
SKIP_INTEGRATION_TESTS=1 pytest python/runtimed/tests/ -v     # Skip integration
RUNTIMED_INTEGRATION_TEST=1 pytest python/runtimed/tests/ -v  # CI mode (spawns daemon)

Browser E2E Tests

Playwright tests in apps/notebook/e2e/ are the preferred E2E surface for notebook UI, runtime, fixture, and renderer behavior. They use the Vite browser relay plus the dev daemon, avoiding the slower Tauri WebDriver app build.

pnpm --filter notebook-ui test:e2e:browser -- --project=chromium
pnpm --filter notebook-ui test:e2e:browser -- --project=chromium apps/notebook/e2e/execute-cell-output.spec.ts
pnpm --filter notebook-ui test:e2e:browser:portless -- --project=chromium

Native E2E Tests

Native WebDriverIO tests should stay focused on Tauri-specific setup: app shell boot, cwd-derived untitled notebook behavior, launcher/bootstrap behavior, and other issues that require the real Tauri process.

Running

cargo xtask e2e build       # Build with WebDriver support (required first)
cargo xtask e2e test        # Native smoke/default run
cargo xtask e2e test-all    # Native smoke plus long-tail Tauri fixtures
cargo xtask e2e test-fixture <notebook> <spec>  # Single fixture test

Adding Tests

Fixture test: Create notebook in crates/notebook/fixtures/audit-test/, create spec in e2e/specs/, add to FIXTURE_SPECS in e2e/wdio.conf.js, add to crates/xtask/src/main.rs, add to CI.

Regular test: Create a spec in e2e/specs/. It is picked up automatically if not in FIXTURE_SPECS.

Helpers (e2e/helpers.js)

wry WebDriver Constraints

• Use data-testid attributes. Text selectors return broken refs.
• Use browser.execute() + browser.waitUntil(). executeAsync() is unsupported.
• Use typeSlowly() for CodeMirror. Fast input drops characters.
• Use browser.execute() for iframe testing. switchToFrame() is broken.

Selectors

data-testid: notebook-toolbar, save-button, add-code-cell-button, add-markdown-cell-button, start-kernel-button, restart-kernel-button, interrupt-kernel-button, run-all-button, deps-toggle, trust-dialog, trust-approve-button, deps-panel, deps-add-input.

data-slot: output-area, ansi-stream-output, ansi-error-output.

Other: [data-cell-type="code"], [data-cell-type="markdown"], .cm-content[contenteditable="true"], iframe[sandbox].

Diagnostics

Collecting

Use env -i for system diagnostics to avoid dev env vars (RUNTIMED_DEV, RUNTIMED_WORKSPACE_PATH) leaking through. env -i also clears PATH, so use the installed nteract link (~/.local/bin by default) and --channel to pick the installation.

# Nightly (system)
env -i HOME=$HOME "$HOME/.local/bin/nteract" --channel nightly diagnostics

# Stable (system)
env -i HOME=$HOME "$HOME/.local/bin/nteract" --channel stable diagnostics

# Dev daemon (no env -i needed)
RUNTIMED_DEV=1 RUNTIMED_WORKSPACE_PATH="$(pwd)" ./target/debug/runt diagnostics

Other system commands follow the same env -i pattern:

env -i HOME=$HOME "$HOME/.local/bin/nteract" --channel nightly daemon status
env -i HOME=$HOME "$HOME/.local/bin/nteract" --channel nightly daemon logs -f
env -i HOME=$HOME "$HOME/.local/bin/nteract" --channel stable ps

Read files from tarball without extracting:

tar xzf <archive>.tar.gz -O doctor.json
tar xzf <archive>.tar.gz -O runtimed.log | grep -i 'error\|panic'

What to Look For

• Ghost windows: Context for '...' missing in notebook.log
• Daemon crashes: Check runtimed.log.1 (previous session)
• Upgrade failures: Search [upgrade] in notebook.log
• Kernel issues: Search [daemon-kernel] or kernel_status
• Sync errors: Search [notebook-sync] or daemon:disconnected
• Frontend errors: webview:error or webview:warn in notebook.log
• launchd issues: Check doctor.json launchd_service status

Test Philosophy

Prefer fast integration tests over slow E2E. Use E2E for critical user journeys, integration tests for daemon behavior, unit tests for algorithms.