In-VM Testing
capsem-doctor
The diagnostic suite runs inside the guest VM via pytest. Tests live in guest/artifacts/diagnostics/ and are baked into the rootfs.
Running diagnostics
just exec "capsem-doctor" # Full suite (~10s total)
just exec "capsem-doctor -k sandbox" # Only sandbox tests
just exec "capsem-doctor -k network" # Only network tests
just exec "capsem-doctor -x" # Stop on first failure
Test categories
| File |
What it verifies |
test_sandbox.py |
Read-only rootfs, binary permissions, setuid/setgid, kernel hardening (no modules, no debugfs, no IPv6, no swap), process integrity, network isolation (dummy0, fake DNS, iptables) |
test_network.py |
MITM CA in system store + certifi, curl without -k, Python urllib HTTPS, CA env vars, HTTP/80 blocked, non-443 blocked, direct IP blocked, multi-domain DNS, AI provider domains |
test_environment.py |
TERM/HOME/PATH env vars, bash shell, kernel version, aarch64 arch, mount points, tmpfs |
test_runtimes.py |
Python3, Node.js, npm, pip3, git version checks, Python/Node file I/O, git workflow |
test_utilities.py |
~36 unix utilities (coreutils, text processing, network, system tools) |
test_workflows.py |
Text write/read, JSON roundtrip, shell pipes, large file (10MB) |
test_ai_cli.py |
claude/gemini/codex installed and executable |
test_virtiofs.py |
VirtioFS mount, ext4 loopback, workspace I/O, pip install, file delete+recreate |
Adding new in-VM tests
- Add test functions to the appropriate
guest/artifacts/diagnostics/test_*.py or create test_<category>.py
- Use
from conftest import run for shell commands, output_dir fixture for temp files
- Tests auto-skip outside the capsem VM (conftest checks for root + writable /root)
- Rebuild rootfs with
just _build-assets to bake new test files into the image
- For fast iteration during development, tests in
diagnostics/ are also repacked into the initrd by just exec, so just exec "capsem-doctor" picks up changes without a full rootfs rebuild
- Verify:
just exec "capsem-doctor -k <your_test>"
Session inspection
After running a VM session, inspect the telemetry database:
python3 build_system/scripts/doctor/check_session.py # Latest session
python3 build_system/scripts/doctor/check_session.py <session-id> # Specific session
python3 build_system/scripts/doctor/check_session.py --list # List recent sessions
python3 build_system/scripts/doctor/check_session.py -n 10 # Show 10 preview rows per table
Checks: session ledgers exist (net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, security_rule_events), row counts, orphaned tool_calls, AI-provider consistency.
Verifying telemetry pipelines
Each pipeline can be tested with a targeted VM command:
- fs_events:
just exec 'touch /root/test.txt && sleep 1' then python3 build_system/scripts/doctor/check_session.py
- net_events:
just exec 'curl -s https://api.anthropic.com/ && sleep 1'
- model_calls/tool_calls: boot interactively, run
claude -p "what is 2+2"
- MCP-origin tool_calls: boot interactively, run
claude -p "use fetch to get https://example.com" and query tool_calls WHERE origin = 'mcp'
If events are missing: check boot logs for daemon startup, vsock connection acceptance, and whether the VM lived long enough for the debouncer to flush (add sleep 1).
Test fixture
The fixture (tests/fixtures/session/test.db) is a real session DB shared by frontend mock mode and Rust roundtrip tests. No synthetic data.
Updating the fixture
# 1. Run integration test to generate a rich session
python3 build_system/scripts/test/integration_test.py --binary cache/target/cargo/debug/capsem --assets assets
# 2. Inspect completeness
python3 build_system/scripts/doctor/check_session.py <session-id>
# 3. Fixture refresh has no Just convenience command. Copy, checkpoint, and
# scrub the selected DB using the procedure in /dev-session-debug.
# 4. Verify
cargo test --workspace
The fixture must contain: both allowed and denied net_events, created/modified/deleted fs_events, model_calls with cost > 0, tool_calls with origin populated.
1---2name: dev-testing-vm3description: In-VM diagnostics and test fixtures. Use when adding in-VM tests, debugging failures inside the guest, or updating the test fixture.4---56# In-VM Testing78## capsem-doctor910The diagnostic suite runs inside the guest VM via pytest. Tests live in `guest/artifacts/diagnostics/` and are baked into the rootfs.1112### Running diagnostics1314```bash15just exec "capsem-doctor" # Full suite (~10s total)16just exec "capsem-doctor -k sandbox" # Only sandbox tests17just exec "capsem-doctor -k network" # Only network tests18just exec "capsem-doctor -x" # Stop on first failure19```2021### Test categories2223| File | What it verifies |24|------|------------------|25| `test_sandbox.py` | Read-only rootfs, binary permissions, setuid/setgid, kernel hardening (no modules, no debugfs, no IPv6, no swap), process integrity, network isolation (dummy0, fake DNS, iptables) |26| `test_network.py` | MITM CA in system store + certifi, curl without -k, Python urllib HTTPS, CA env vars, HTTP/80 blocked, non-443 blocked, direct IP blocked, multi-domain DNS, AI provider domains |27| `test_environment.py` | TERM/HOME/PATH env vars, bash shell, kernel version, aarch64 arch, mount points, tmpfs |28| `test_runtimes.py` | Python3, Node.js, npm, pip3, git version checks, Python/Node file I/O, git workflow |29| `test_utilities.py` | ~36 unix utilities (coreutils, text processing, network, system tools) |30| `test_workflows.py` | Text write/read, JSON roundtrip, shell pipes, large file (10MB) |31| `test_ai_cli.py` | claude/gemini/codex installed and executable |32| `test_virtiofs.py` | VirtioFS mount, ext4 loopback, workspace I/O, pip install, file delete+recreate |3334### Adding new in-VM tests35361. Add test functions to the appropriate `guest/artifacts/diagnostics/test_*.py` or create `test_<category>.py`372. Use `from conftest import run` for shell commands, `output_dir` fixture for temp files383. Tests auto-skip outside the capsem VM (conftest checks for root + writable /root)394. Rebuild rootfs with `just _build-assets` to bake new test files into the image405. For fast iteration during development, tests in `diagnostics/` are also repacked into the initrd by `just exec`, so `just exec "capsem-doctor"` picks up changes without a full rootfs rebuild416. Verify: `just exec "capsem-doctor -k <your_test>"`4243## Session inspection4445After running a VM session, inspect the telemetry database:4647```bash48python3 build_system/scripts/doctor/check_session.py # Latest session49python3 build_system/scripts/doctor/check_session.py <session-id> # Specific session50python3 build_system/scripts/doctor/check_session.py --list # List recent sessions51python3 build_system/scripts/doctor/check_session.py -n 10 # Show 10 preview rows per table52```5354Checks: session ledgers exist (net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, security_rule_events), row counts, orphaned tool_calls, AI-provider consistency.5556## Verifying telemetry pipelines5758Each pipeline can be tested with a targeted VM command:5960- **fs_events**: `just exec 'touch /root/test.txt && sleep 1'` then `python3 build_system/scripts/doctor/check_session.py`61- **net_events**: `just exec 'curl -s https://api.anthropic.com/ && sleep 1'`62- **model_calls/tool_calls**: boot interactively, run `claude -p "what is 2+2"`63- **MCP-origin tool_calls**: boot interactively, run `claude -p "use fetch to get https://example.com"` and query `tool_calls WHERE origin = 'mcp'`6465If events are missing: check boot logs for daemon startup, vsock connection acceptance, and whether the VM lived long enough for the debouncer to flush (add `sleep 1`).6667## Test fixture6869The fixture (`tests/fixtures/session/test.db`) is a real session DB shared by frontend mock mode and Rust roundtrip tests. No synthetic data.7071### Updating the fixture7273```bash74# 1. Run integration test to generate a rich session75python3 build_system/scripts/test/integration_test.py --binary cache/target/cargo/debug/capsem --assets assets7677# 2. Inspect completeness78python3 build_system/scripts/doctor/check_session.py <session-id>7980# 3. Fixture refresh has no Just convenience command. Copy, checkpoint, and81# scrub the selected DB using the procedure in /dev-session-debug.8283# 4. Verify84cargo test --workspace85```8687The fixture must contain: both allowed and denied net_events, created/modified/deleted fs_events, model_calls with cost > 0, tool_calls with origin populated.
Google