capsem-doctor
capsem-doctor is a pytest-based diagnostic suite that runs inside the guest VM. It verifies sandbox integrity, network isolation, runtime environment, and AI agent functionality. It's the smoke test gate -- every change must pass it before shipping.
Doctor is also an Ironbank input. When doctor is used to close a
release-critical VM/security/protocol/package-manager gate, load /ironbank
and assert the full ledger through tests/ironbank/: client result, DB rows,
structured logs, UDS/HTTP route output, counters, and UI-facing JSON. A doctor
exit code or "row exists" check is not enough.
Running
just exec "capsem-doctor" # Full suite (~10s total including VM boot)
just exec "capsem-doctor -k sandbox" # Only sandbox tests
just exec "capsem-doctor -k network" # Only network tests
just exec "capsem-doctor -x" # Stop on first failure
just exec "capsem-doctor -v" # Extra verbose
Test categories (11 files)
| File |
What it validates |
test_sandbox.py |
Read-only rootfs, binary permissions (chmod 555), no setuid/setgid, kernel hardening (no modules, no debugfs, no IPv6, no swap, no kallsyms), process integrity (pty-agent, dnsmasq running; no systemd, sshd, cron), network isolation (dummy0, fake DNS, iptables, no real NICs) |
test_network.py |
MITM CA in system store + certifi, curl without -k works, Python urllib HTTPS, CA env vars set (SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS), HTTP/80 blocked, non-443 ports blocked, direct IP blocked, multi-domain DNS faking, AI provider domains reachable |
test_environment.py |
TERM/HOME/PATH env vars correct, shell is bash, kernel version, aarch64 arch, mount points (/proc, /sys, /dev, /dev/pts), tmpfs verification |
test_runtimes.py |
Python3, Node.js, npm, pip3, git version checks; Python file I/O; Node file I/O; git init+commit workflow |
test_utilities.py |
~36 unix utilities available (coreutils, text processing, network, system tools, capsem-bench) |
test_workflows.py |
Text write/read, JSON roundtrip (Python + Node), shell pipes, large file (10MB) |
test_ai_cli.py |
claude, gemini, codex installed and executable without crashing |
test_virtiofs.py |
VirtioFS root mount, ext4 loopback upper, loop device active, workspace write/read/large file/subdir, system overlay writable, pip install works, file delete+recreate (skipped in block mode) |
test_mcp.py |
Guest MCP endpoint tool routing, domain blocking via MCP |
test_injection.py |
Security injection tests |
conftest.py |
Test infrastructure (auto-skip outside VM, run() helper, output dir fixture) |
Infrastructure (conftest.py)
# Auto-skip if not in capsem VM (checks root + writable /root)
def pytest_ignore_collect(collection_path, config):
if os.geteuid() != 0 or not os.access("/root", os.W_OK):
return True
# Shell command runner
def run(cmd, timeout=10):
return subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
# Shared output directory: /root/tests
@pytest.fixture
def output_dir():
return TESTS_OUTPUT_DIR
Adding a new test
- Add test functions to the appropriate
guest/artifacts/diagnostics/test_*.py file, or create test_<category>.py
- Use
from conftest import run for shell commands, output_dir fixture for temp files
- Tests auto-skip outside the capsem VM (no special guards needed)
just exec "capsem-doctor" picks up changes immediately (diagnostics repacked into initrd)- For rootfs-baked changes:
just _build-assets then just exec "capsem-doctor"
Where tests live on disk
- Source:
guest/artifacts/diagnostics/test_*.py (in the repo)
- In rootfs:
/usr/local/lib/capsem-tests/test_*.py (baked by Dockerfile.rootfs)
- In initrd: overrides rootfs copies via
_pack-initrd (fast iteration)
Writing good diagnostic tests
- Test one thing per function. Name clearly:
test_readonly_rootfs, test_ca_in_certifi
- Use
run() for shell commands, check .returncode and .stdout/.stderr
- Set reasonable timeouts (default 10s). Network tests may need longer.
- Think adversarially: test what should be blocked, not just what should work
- For VirtioFS tests, skip gracefully in block mode:
pytest.mark.skipif
- For Ironbank/release gates, do not skip. If a package manager, protocol, or
diagnostic dependency is unavailable, the product or harness is broken.
- Package-manager diagnostics must prove function, not installation presence:
run the installed binary/module and verify deterministic behavior.
1---2name: dev-capsem-doctor3description: The capsem-doctor in-VM diagnostic suite. Use when writing, running, or extending doctor tests, or debugging VM sandbox issues.4---56# capsem-doctor78capsem-doctor is a pytest-based diagnostic suite that runs inside the guest VM. It verifies sandbox integrity, network isolation, runtime environment, and AI agent functionality. It's the smoke test gate -- every change must pass it before shipping.910Doctor is also an Ironbank input. When doctor is used to close a11release-critical VM/security/protocol/package-manager gate, load `/ironbank`12and assert the full ledger through `tests/ironbank/`: client result, DB rows,13structured logs, UDS/HTTP route output, counters, and UI-facing JSON. A doctor14exit code or "row exists" check is not enough.1516## Running1718```bash19just exec "capsem-doctor" # Full suite (~10s total including VM boot)20just exec "capsem-doctor -k sandbox" # Only sandbox tests21just exec "capsem-doctor -k network" # Only network tests22just exec "capsem-doctor -x" # Stop on first failure23just exec "capsem-doctor -v" # Extra verbose24```2526## Test categories (11 files)2728| File | What it validates |29|------|-------------------|30| `test_sandbox.py` | Read-only rootfs, binary permissions (chmod 555), no setuid/setgid, kernel hardening (no modules, no debugfs, no IPv6, no swap, no kallsyms), process integrity (pty-agent, dnsmasq running; no systemd, sshd, cron), network isolation (dummy0, fake DNS, iptables, no real NICs) |31| `test_network.py` | MITM CA in system store + certifi, curl without -k works, Python urllib HTTPS, CA env vars set (SSL_CERT_FILE, REQUESTS_CA_BUNDLE, NODE_EXTRA_CA_CERTS), HTTP/80 blocked, non-443 ports blocked, direct IP blocked, multi-domain DNS faking, AI provider domains reachable |32| `test_environment.py` | TERM/HOME/PATH env vars correct, shell is bash, kernel version, aarch64 arch, mount points (/proc, /sys, /dev, /dev/pts), tmpfs verification |33| `test_runtimes.py` | Python3, Node.js, npm, pip3, git version checks; Python file I/O; Node file I/O; git init+commit workflow |34| `test_utilities.py` | ~36 unix utilities available (coreutils, text processing, network, system tools, capsem-bench) |35| `test_workflows.py` | Text write/read, JSON roundtrip (Python + Node), shell pipes, large file (10MB) |36| `test_ai_cli.py` | claude, gemini, codex installed and executable without crashing |37| `test_virtiofs.py` | VirtioFS root mount, ext4 loopback upper, loop device active, workspace write/read/large file/subdir, system overlay writable, pip install works, file delete+recreate (skipped in block mode) |38| `test_mcp.py` | Guest MCP endpoint tool routing, domain blocking via MCP |39| `test_injection.py` | Security injection tests |40| `conftest.py` | Test infrastructure (auto-skip outside VM, `run()` helper, output dir fixture) |4142## Infrastructure (conftest.py)4344```python45# Auto-skip if not in capsem VM (checks root + writable /root)46def pytest_ignore_collect(collection_path, config):47 if os.geteuid() != 0 or not os.access("/root", os.W_OK):48 return True4950# Shell command runner51def run(cmd, timeout=10):52 return subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)5354# Shared output directory: /root/tests55@pytest.fixture56def output_dir():57 return TESTS_OUTPUT_DIR58```5960## Adding a new test61621. Add test functions to the appropriate `guest/artifacts/diagnostics/test_*.py` file, or create `test_<category>.py`632. Use `from conftest import run` for shell commands, `output_dir` fixture for temp files643. Tests auto-skip outside the capsem VM (no special guards needed)654. `just exec "capsem-doctor"` picks up changes immediately (diagnostics repacked into initrd)665. For rootfs-baked changes: `just _build-assets` then `just exec "capsem-doctor"`6768## Where tests live on disk6970- **Source**: `guest/artifacts/diagnostics/test_*.py` (in the repo)71- **In rootfs**: `/usr/local/lib/capsem-tests/test_*.py` (baked by Dockerfile.rootfs)72- **In initrd**: overrides rootfs copies via `_pack-initrd` (fast iteration)7374## Writing good diagnostic tests7576- Test one thing per function. Name clearly: `test_readonly_rootfs`, `test_ca_in_certifi`77- Use `run()` for shell commands, check `.returncode` and `.stdout`/`.stderr`78- Set reasonable timeouts (default 10s). Network tests may need longer.79- Think adversarially: test what should be blocked, not just what should work80- For VirtioFS tests, skip gracefully in block mode: `pytest.mark.skipif`81- For Ironbank/release gates, do not skip. If a package manager, protocol, or82 diagnostic dependency is unavailable, the product or harness is broken.83- Package-manager diagnostics must prove function, not installation presence:84 run the installed binary/module and verify deterministic behavior.
Google