# Skill > smolBSD — Complete Platform Reference - Skill: `netbsdfr/skill` (Agent Skill) - Install (CLI): `npx skillmds@latest add netbsdfr/skill` - Raw SKILL.md: https://api.skillmd.com/api/skills/netbsdfr/skill/raw - Safety review: pending (external: skill-scanner WARNING, skillspector FAIL) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: netbsdfr (https://skillmd.com/u/netbsdfr) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/netbsdfr/skill --- # smolBSD — Complete Platform Reference This skill provides exhaustive knowledge of the entire smolBSD framework — enough for an agent to understand, navigate, extend, and debug every aspect of the project. **Read §20 (Debugging Playbook) before touching anything related to image minimization, sailor, or login/PAM in a stripped image** — these are the areas that have produced the most subtle real-world failures and are easy to misdiagnose from source alone. ## 1. Project Overview **smolBSD** builds minimal, fast-booting NetBSD virtual machines (microVMs). Key properties: - **~10 ms boot** via PVH (PVHv2) on QEMU `microvm` machine type - **No prior NetBSD installation** required on the host - **Immutable by design** — images are built once, booted many times - **Host platforms**: GNU/Linux, NetBSD, macOS (x86 VT-capable or ARM64 CPU recommended) - **Guest architectures**: `amd64`, `i386`, `evbarm-aarch64` - **VMMs**: QEMU (primary), Firecracker, Bhyve (BIOS mode) - **Images** are raw `.img` disk files with FFS (NetBSD) or ext2 (Linux build hosts) The fundamental unit is a **service** — a directory containing: - NetBSD set selection (`base`, `etc`, `comp`, `man`, `rescue`, …) - Build-time scripts (`postinst/*.sh`) - Runtime init script (`etc/rc`) - Build configuration (`options.mk`) --- ## 2. Project Directory Layout ``` smolBSD/ ├── Makefile # Entry point for manual image building (bmake) ├── mkimg.sh # Image creation script (called by Makefile, or directly via `bmake SERVICE= base`) ├── startnb.sh # Low-level QEMU VM launcher ├── smoler.sh # High-level CLI dispatcher: build|run|push|pull|images ├── batch.sh # Batch launcher: N copies of a service on shifted ports ├── smoler/ │ ├── build.sh # SMOLerfile parser → generates service dir + calls bmake │ └── img.sh # OCI push/pull/list (oras wrapper) ├── scripts/ │ ├── app-run.sh # Launcher helper for the app/ GUI │ ├── fetch.sh # Smart curl wrapper (globbing, fresh checks) │ ├── freshchk.sh # Freshness check (remote Last-Modified cache in db/) │ ├── sh # Static shell used as /rescue/sh in rescue images │ └── uname.sh # Architecture/machine detection helper ├── sailor/ # Cloned sailor repo (minimization), invoked by mkimg.sh ├── service/ # All service definitions │ ├── common/ # Shared runtime scripts bundled into /etc/include/ in VM │ │ ├── basicrc # Standard env, networking, devices, SSL_CERT_FILE, rc.pre/rc.local │ │ ├── choupi # Emoji/ASCII toggle for terminal output │ │ ├── funcs # rsynclite() helper (tar-based directory sync) │ │ ├── vars # BASEPATH, DRIVE2 path constants │ │ ├── shutdown # Clean halt (sync, umount, optional viocon kill signal) │ │ ├── mount9p # 9P filesystem mount (host directory sharing) │ │ ├── qemufwcfg # QEMU fw_cfg variable loader │ │ ├── pkgin # Package manager bootstrapper │ │ └── sailor.vars # Sailor integration variables │ ├── base/ # Full base+etc system with ksh (builder image base) │ ├── build/ # Builder microVM service (orchestrates service builds) │ ├── rescue/ # ~10 MB minimal rescue shell │ └── / # One directory per service │ ├── etc/rc # Runtime init script (MANDATORY for init(8) services) │ ├── postinst/ # Build-time scripts executed on host/VM builder │ ├── options.mk # Service build variables (IMGSIZE, ADDPKGS, SETS, etc.) │ ├── own.mk # User overrides (git-ignored, not committed) │ ├── sailor.conf # Sailor minimization rules │ ├── packages/ # Pre-built binary packages for offline install │ └── NETBSD_ONLY # Marker: build only on native NetBSD ├── smolerfiles/ # SMOLerfile / Dockerfile examples │ ├── Dockerfile.inc # Shared INCLUDE snippets │ ├── Dockerfile. # Per-service SMOLerfile (Dockerfile-compatible) │ ├── SMOLerfile. # Named SMOLerfiles (same syntax, different naming) │ ├── *.smol # Minimal SMOLerfiles (service name from filename) │ └── *.inc # Shared include fragments ├── etc/ # VM config files for startnb.sh (-f flag) │ └── .conf # hostfwd, imgtag, use_pty, KERNEL, NBIMG, etc. ├── bios/ # BIOS firmware files for microvm machine type ├── confkerndev/ # Kernel driver disabler tool (SMOLIFY) ├── app/ # Flask-based web GUI for VM management ├── www/ # Project website and assets ├── k8s/ # Kubernetes device plugin / deployment examples ├── misc/ # Miscellaneous documentation ├── contribs/ # Contributed scripts ├── share/ # Shared assets (e.g. ssh.pub keys) ├── .github/workflows/ # CI/CD pipeline │ ├── main.yml # Builder + rescue images for amd64 + evbarm-aarch64 on push │ └── smoler.yml # SMOLerfile service images on smolerfiles/* push ├── db/ # Fetch freshness cache (remote Last-Modified, see freshchk.sh) ├── images/ # Built .img disk images (empty in repo, populated at build) ├── kernels/ # Downloaded kernels (empty in repo, populated at build) ├── sets/ # Downloaded NetBSD sets (empty in repo, populated at build) ├── pkgs/ # Optional pre-fetched packages (empty in repo, populated at build) ├── mnt/ # Build-time mount point (empty directory) └── disks/ # Additional disk images ``` --- ## 3. Two Workflows ### 3.1 smoler.sh (Docker-style, high-level) `smoler.sh` is a thin dispatcher that routes subcommands to dedicated scripts: | Command | Routes To | Purpose | |---------|-----------|---------| | `./smoler.sh build [-y] [-t tag] [--build-arg K=V] [VAR=val] ` | `smoler/build.sh` | Parse SMOLerfile → generate service dir → call `bmake build` | | `./smoler.sh run [startnb.sh flags]` | `startnb.sh` | Run a built image (resolves name → config file or raw path). **Any** `startnb.sh` flag passes through after the image name (e.g. `-l drive2,drive3` for extra drives, `-r`, `-e`). | | `./smoler.sh push ` | `smoler/img.sh` | Push to OCI registry via oras | | `./smoler.sh pull ` | `smoler/img.sh` | Pull from OCI registry via oras | | `./smoler.sh images [ok]` | `smoler/img.sh` | List local images with size, date, signature status. Columns are sized as fixed proportions of terminal width (name = 50%, size/date/sig = 25% each); not auto-fit to content. Pass `ok` to show only images with verified `smolsig`. | **`smoler.sh run` name resolution:** 1. Strips `-amd64:…` or `-evbarm-aarch64:…` suffix to get base service name (note: an `-i386:…` suffix is **not** stripped — the `etc/.conf` lookup for i386 images is a known quirk; the `images/.img` fallback still works with the full name) 2. Checks for `etc/.conf` → passes `-f etc/.conf` to `startnb.sh` 3. Falls back to checking `images/.img` → passes `-i ` to `startnb.sh` 4. If neither exists, shows `startnb.sh -h` usage **`smoler.sh build` regeneration:** if `service//` already exists, build.sh deletes its `etc/rc`, `options.mk`, `postinst/` and `etc/.conf` before regenerating them. Untracked files in the service dir (e.g. `sailor.conf`, `own.mk`) survive, but anything hand-edited in the deleted files is lost — commit what matters first. ### 3.2 bmake / make (Manual, low-level) | Command | Purpose | |---------|---------| | `bmake buildimg` | Build the builder image (native on NetBSD/FreeBSD/Linux; on macOS this fails — the `build` target falls back to `fetchimg` there) | | `bmake fetchimg` | Download pre-built builder image from GitHub Releases (macOS, no FFS support) | | `bmake SERVICE= build` | Build a service image using the builder microVM | | `bmake SERVICE= base` | Build only the base filesystem (no builder VM — runs `mkimg.sh` directly) | | `bmake SERVICE= MOUNTRO=y build` | Build with read-only root | | `bmake SERVICE= ARCH=evbarm-aarch64 build` | Build for ARM64 | | `bmake kernfetch` | Download the appropriate kernel | | `bmake setfetch` | Download NetBSD sets | | `bmake pkgfetch` | Download binary packages | | `bmake fetchall` | All of the above | | `bmake rescue` | Shortcut: `SERVICE=rescue build` | | `bmake live` | Fetch a full NetBSD live image | **Platform-specific builder image behavior (Makefile `build` target):** - On **NetBSD/FreeBSD/Linux**: builds the builder image natively (`bmake buildimg`) - On **macOS** (and any other OS): fetches the pre-built builder image from GitHub (`bmake fetchimg`); running `buildimg` directly on macOS fails because mkimg.sh rejects macOS - Builder image freshness is checked via SHA256; rebuilds/fetches only when the remote changes --- ## 4. SMOLerfile / Dockerfile Reference SMOLerfiles are nearly 100% Dockerfile-compatible. `smoler/build.sh` parses them line-by-line and generates: - `service//options.mk` — build variables - `service//etc/rc` — runtime init script - `service//postinst/postinst-N.sh` — build-time execution scripts - `etc/.conf` — VM config for `startnb.sh` ### 4.1 Parsing Flow (build.sh internals) 1. **INCLUDE expansion**: `INCLUDE ` directives are resolved first by catting the referenced file inline, producing a flat temporary SMOLerfile 2. **LABEL extraction**: All `LABEL` lines (with or without `smolbsd.` prefix) are extracted via `sed`/`awk`, uppercased, and written to `options.mk` 3. **Service name**: From `LABEL smolbsd.service=NAME`, or from `.smol` filename (`SMOLerfile.foo` → `SERVICE=foo`) 4. **Postinst-0.sh**: Generated with chroot setup (pkgin bootstrap, resolv.conf, openssl certs) 5. **Line-by-line parsing**: Each directive generates shell commands appended to postinst scripts or `etc/rc` 6. **Finalization**: `etc/rc` gets `. /etc/include/shutdown` appended; `etc/.conf` gets `imgtag` and `use_pty` 7. **Build**: Calls `make` (NetBSD) or `bmake` (elsewhere) with `SERVICE= IMGTAG=: build` ### 4.2 All Supported Directives | Directive | Syntax | Description | |-----------|--------|-------------| | `FROM` | `FROM base,etc` or `FROM base-amd64.img` | Set names or an existing image name. If omitted, the Makefile `SETS` default (`base,etc`) is used. | | `LABEL smolbsd.service=NAME` | `LABEL smolbsd.service=caddy` | **Mandatory.** Sets the service name. | | `LABEL smolbsd.imgsize=N` | `LABEL smolbsd.imgsize=2048` | Image size in MB (default: 512). | | `LABEL smolbsd.minimize=y` | `LABEL smolbsd.minimize=y` | Shrink to actual usage + 10%. `MINIMIZE=+N` adds N MB instead. See §13 and §20.1 before combining with WAPBL. | | `LABEL smolbsd.publish="H:G"` | `LABEL smolbsd.publish="8881:8880,2289:22"` | Port mappings (host:guest), comma-separated. | | `LABEL smolbsd.use_pty=y` | `LABEL smolbsd.use_pty=y` | Use PTY console (needed for interactive apps like vim/tmux). | | `LABEL smolbsd.addpkgs="pkg1 pkg2"` | `LABEL smolbsd.addpkgs="pkgin curl"` | Packages to fetch/untar at build time (no pkgin needed). | | `RUN` | `RUN pkgin up && pkgin -y in caddy` | Execute commands during build (chrooted). Supports heredocs (`<> /etc/rc.local EOF ``` The parser detects `<.conf Format ```sh hostfwd=::8881-:8880,::2289-:22 imgtag=latest use_pty=y share=/host/path # from VOLUME ``` ### 4.7 Postinst Script Numbering The parser generates numbered postinst scripts: - `postinst-0.sh` — chroot bootstrap (pkgin setup, resolv.conf, openssl certs) - `postinst-1.sh` — first RUN/COPY/ADD/USER/VOLUME/WORKDIR block (default shell) - `postinst-N.sh` — new script created when `SHELL` directive changes the shell - `postinst.args` — accumulated ARG/ENV exports shared across scripts ### 4.8 File Naming Conventions - **`Dockerfile.`** — standard Dockerfile naming; service name from `LABEL smolbsd.service` - **`SMOLerfile.`** — same syntax; service name from `LABEL smolbsd.service` - **`.smol`** — minimal files; service name extracted from filename itself - **`*.inc`** — include fragments (used with `INCLUDE` directive) ### 4.9 make vs bmake `bmake` is the **host-side** build tool (required on Linux/macOS; on NetBSD it's synonymous with `make`). It invokes the top-level `Makefile` targets (`build`, `buildimg`, `base`, …). Inside the **builder VM** (i.e., in `RUN` directives and `postinst/*.sh` scripts), the environment is NetBSD — use plain `make`, not `bmake`. The builder VM includes `make` from the `comp` set; `bmake` is not guaranteed to be available. ```dockerfile # Wrong (bmake is a host tool, not inside the VM): RUN cd /tmp/src && bmake && bmake install # Correct (plain make inside the NetBSD builder VM): RUN cd /tmp/src && make && make install ``` --- ## 5. Service Directory Manual Reference ### 5.1 options.mk — All Known Variables | Variable | Type | Default | Description | |----------|------|---------|-------------| | `SERVICE` | string | (target name) | Service name, determines output filename | | `IMGSIZE` | int | 512 | Image size in megabytes | | `SETS` | string | `base.${SETSEXT} etc.${SETSEXT}` | NetBSD sets to include (space-separated) | | `ADDSETS` | string | (empty) | Additional sets beyond SETS | | `ADDPKGS` | string | (empty) | Packages to fetch and extract into image | | `MINIMIZE` | y/+N | (empty) | `y` = +10%, `+512` = explicit MB to add | | `MOUNTRO` | y | (empty) | Mount root read-only (`-o` passed to mkimg.sh) | | `BIOSBOOT` | y | (empty) | Enable BIOS boot (GPT + bootxx_ffsv1) | | `BIOSCONSOLE` | string | `com0` | Console device for BIOS boot (`com0`, `pc`) | | `SMOLIFY` | y | (empty) | Run confkerndev to disable unused kernel drivers | | `FROMIMG` | string | (empty) | Inherit from existing image instead of sets | | `PKGVERS` | string | `11.0` | Package version for pkgsrc URL | | `ARCH` | string | (detected) | Target architecture: `amd64`, `i386`, `evbarm-aarch64` | | `CURLSH` | string | (empty) | URL to a shell script executed as finalizer | | `SETSEXT` | string | `tar.xz` | Set archive extension (`tgz` for i386) | | `IMGTAG` | string | (empty) | Suffix appended to image name (e.g. `:latest`) | | `SVCIMG` | string | (empty) | When set, only run `postinst/.sh` | | `PUBLISH` | string | (empty) | Port mappings (used by SMOLerfile parser for EXPOSE) | **Conditional variables** (Makefile syntax in options.mk): ```makefile .if defined(MINIMIZE) && ${MINIMIZE} == y ADDPKGS=pkgin pkg_tarup pkg_install sqlite3 rsync curl .endif ``` ### 5.2 etc/rc — Runtime Init Script This is the heart of every service. Standard structure: ```sh #!/bin/sh . /etc/include/basicrc # Mandatory: env, networking, devices . /etc/include/mount9p # Optional: host directory sharing # tmpfs mounts for writable overlays mount -t tmpfs -o -s10M tmpfs /tmp mount -t tmpfs -o -s10M tmpfs /var/log mount -t tmpfs -o -s1M tmpfs /var/run mount -t tmpfs -o -s10M -o union tmpfs /etc # Service-specific setup (users, permissions, config) useradd -m sshd mkdir -p /home/sshd/.ssh # Start services /etc/rc.d/sshd onestart # Main command (blocks until service exits) exec myapp . /etc/include/shutdown # Clean halt ``` **Key hooks in basicrc:** - `/etc/rc.pre` — custom pre-boot hook (sourced before device setup) - `/etc/rc.local` — custom post-boot hook (sourced after networking, before MOUNTRO) - `SSL_CERT_FILE` env var — if set, copies custom SSL certs and runs `certctl rehash` ### 5.3 postinst/*.sh — Build-Time Scripts These execute **on the build host** (or builder VM) inside the mounted image root. Use for: - Downloading external binaries with `curl` or `ftp` - Extracting archives - Setting up chroot environment - Pre-configuration that doesn't need pkgin They are NOT run inside the microVM at boot time. **Important conventions:** - Scripts run from the **mounted image root** (i.e., `pwd` is the fake root) - Paths like `etc/ssh/` refer to the image's `/etc/ssh/` - Use `../service//etc/` to access files from the service directory - Source `../service/common/funcs` for `rsynclite()` and `../service/common/choupi` for emoji output - Check `/BUILDIMG` marker file to verify running inside the builder VM ### 5.4 own.mk — User Overrides Not committed to git. Same format as `options.mk`. Loaded **after** `options.mk` so it overrides. Use for personal dev settings. ```makefile # service/myapp/own.mk (git-ignored) IMGSIZE=1024 ADDPKGS=pkgin curl vim ``` ### 5.5 sailor.vars (in service/common/) Seed variables consumed by sailor when `mkimg.sh` runs minimization (see §13.2). Current contents: - `shippath` — the smolBSD build drive path (`/drive2`), where sailor finds the image being minimized - `shipbins` — baseline list of binaries always kept (init, mount, sh, useradd, login, `/usr/lib/security/*`, …) - `sync_dirs` — directories kept in sync rather than stripped (`/etc`, certs, pkgin config, terminfo, zoneinfo, `/var/log`) - `packages` — package names treated as ship targets (`curl`, `rsync`) **Relationship:** treat `sailor.vars` as the **floor**, and per-service `sailor.conf` as the **diff** on top of it. If a stripped image later fails in surprising ways (WAPBL errors, missing PAM modules, broken `login`), the fix is almost always to add a keep-rule in `sailor.vars` or the service's own `sailor.conf`, not to patch `mkimg.sh` — see §20. ### 5.6 packages/ — Offline Binary Packages Place pre-built `.tgz` packages here. `mkimg.sh` rsyncs them to the image root as `/packages/`. The `pkgin` common script detects `/packages/` and installs them via `pkg_add`. ### 5.7 NETBSD_ONLY — Platform Marker If this empty file exists, `mkimg.sh` refuses to build on non-NetBSD hosts: ``` This image must be built on NetBSD! Use the image builder instead: make SERVICE= build ``` --- ## 6. Build Pipeline — Deep Dive ### 6.1 Image Creation (bmake SERVICE=foo build) The `build` target in the Makefile orchestrates a two-stage process: **Stage 1: Builder microVM creation** ``` bmake buildimg ``` 1. `SERVICE=build IMGTAG= base` — calls `mkimg.sh` to create `images/build-amd64.img` 2. Extracts `base` + `etc` sets (plus partial `comp:/usr/bin/strip`) with `MOUNTRO=y` 3. Creates FFS (NetBSD) or ext2 (Linux) filesystem on the image 4. Installs the builder's own `/etc/rc` that waits for a second drive and executes build commands **Stage 2: Service build inside builder VM** ``` bmake SERVICE=foo build ``` 1. `fetchall` — download sets, packages, and kernel 2. Creates a blank disk image of `IMGSIZE` MB (via `dd`) 3. Writes `ENVVARS` to `tmp/build-foo` (lock/coordination file) 4. Launches the builder VM with `startnb.sh`: - `-k kernels/netbsd-SMOL` — PVH kernel - `-i images/build-amd64.img` — builder rootfs - `-l images/foo-amd64.img` — second drive (target image; `-l` also accepts a comma-separated list for multiple extra drives) - `-w .` — 9P share of project directory - `-p ::22022-:22` — SSH access - `-c $BUILDCPUS -m $BUILDMEM` (defaults 2 cores / 1024 MB) - `-x "-pidfile qemu-.pid"` 5. Builder VM's `/etc/rc` detects the second drive, sources `tmp/build-foo`, calls `make base` to invoke `mkimg.sh` to populate the target image 6. Builder removes `tmp/build-foo` when done (a final `cat` keeps the VM alive after) 7. Host polls the lock file, then kills builder QEMU via the pidfile 8. If `MINIMIZE` is set, waits for the image to be released (`lsof`), then resizes via `qemu-img resize --shrink $(cat tmp/.size)` — **if the image also uses WAPBL journaling, see §20.1** 9. Writes signature to image and `.sig` file: `smolsig:DD/MM/YYYY|UUID` ### 6.2 mkimg.sh — Internal Flow 1. Source `tmp/build-*` for ENVVARS (SERVICE, ARCH, PKGVERS, etc.) 2. Source `service/common/vars`, `funcs`, `choupi` 3. Detect OS: NetBSD (a smolBSD builder image reports `smolBSD` — both accepted), Linux (ext2 path), FreeBSD (gpart/mdconfig path). OpenBSD, macOS and anything else **exit 1**. `MINIMIZE` or a `NETBSD_ONLY` marker also aborts on non-NetBSD hosts. 4. If `FROMIMG` is set, copy existing image (or `dd` onto the secondary disk); otherwise `dd` zero-filled image (host path only) 5. **Partition and format:** - **Linux**: `sgdisk` + `losetup` + `mke2fs -O none` (ext2, no journal) - **FreeBSD**: `gpart` + `mdconfig` + `newfs` (FFS) - **NetBSD**: `gpt` + `vndconfig`/`dkctl` wedge lookup + `newfs` (FFS). WAPBL log is only enabled when **not** MINIMIZE (`noatime` always; this is the guard behind §20.1) 6. Extract ADDPKGS packages into `${LOCALBASE}` (e.g., `/usr/pkg`); exception: with sailor minimization requested, packages are cleanly reinstalled via `pkgin` from `/tmp/usrpkg.tgz` (backed up by the builder VM's `/etc/rc`) 7. If `MINIMIZE` + `sailor.conf` exists + `/var/db/pkgin` present: run sailor — `cd ${BASEPATH}/sailor && ./sailor.sh build /service//sailor.conf` (seeded by `service/common/sailor.vars`, see §5.5) 8. Extract sets (`tar xfp`) — supports partial extraction (`set:path`); or copy a hand-made `rootdir` tar 9. **Write `/etc/fstab` atomically** (single full-content write): `NAME=root / 1 1` — **see §20.2 for a real corruption bug if this step is touched** 10. Rsync `service//etc/` → mounted `/etc/` 11. Rsync `service/common/` → mounted `/etc/include/` 12. Rsync `service//packages/` → mounted `/` (as `/packages/`) 13. Copy kernel if specified (`-k`, to `/netbsd`) 14. **cd into mounted root**; rescue symlink shims; run `postinst/*.sh` scripts sequentially (sorted by `ls`, `sh $x`, `SVCIMG` filter) 15. On non-NetBSD: backup `MAKEDEV` to `etc/`, patch `unionfs` out of `dev/MAKEDEV` (atomic `mv`) 16. Write `PKGVERS` to `etc/pkgvers` 17. If `CURLSH` set: `curl -sSL | /bin/sh` 18. If `MINIMIZE`: clean `/var/db/pkgin` 19. Create `/var/qemufwcfg` mount point 20. If BIOS boot: copy `/usr/mdec/boot`, create `boot.cfg` 21. Unmount; if `MINIMIZE`: `du -s` + `resize_ffs -y -s` + `fsck_ffs -c4 -f -y`, write new size in bytes to `tmp/.size` 22. Detach loopback/vnd; if BIOS boot (NetBSD only): `gpt biosboot -i 1` + `installboot /usr/mdec/bootxx_ffsv1` **Mount point selection:** - Host build (no secondary disk): `mnt/` directory in project root - Builder VM (secondary disk): `/drive2` (from `service/common/vars`) ### 6.3 Builder VM (service/build/) The builder VM is a special service that: 1. Sources `basicrc` and `mount9p` for networking and host sharing 2. Sets up SSL certificates for HTTPS fetching (tmpfs `/etc/openssl` + `certctl rehash`) 3. Sources `tmp/build-*` to get the target service's build variables 4. If `MINIMIZE` + `sailor.conf`: mounts tmpfs on `/var/db` and `/usr/pkg`, backs up `/usr/pkg` to `/tmp/usrpkg.tgz` (consumed by mkimg.sh's sailor path), and warns if the `sailor/` clone is missing 5. Calls `make base` to invoke `mkimg.sh` for the target service (`ADDPKGS` is excluded from the make invocation) 6. Removes `tmp/build-*` when done (signals the host to kill the VM), then waits on `cat` --- ## 7. Boot / Runtime Pipeline — Deep Dive ### 7.1 startnb.sh — VM Launcher **Key flags:** | Flag | Argument | Description | |------|----------|-------------| | `-f` | config file | Load VM config (sources the file; may define kernel, img, hostfwd, imgtag, use_pty, KERNEL, NBIMG, …) | | `-k` | kernel path | Kernel to boot (defaults by arch) | | `-i` | image path | Root disk image path | | `-I` | (none) | Load image as initrd instead of disk | | `-c` | N | Number of CPU cores (default: 1) | | `-m` | MB | Memory in MB (default: 256) | | `-r` | root name | Root disk wedge name for the `root=` kernel parameter (default: `NAME=root`) | | `-l` | drive2,drive3,… | Extra drives (beyond root) passed to the guest as virtio-blk devices. Comma-separated list, and repeated `-l` flags accumulate. Each drive gets a unique `hd-N` id. Mount points inside the guest are up to the service (`/drive2` etc., see `service/common/vars`) | | `-p` | ports | Port forwarding: `[tcp|udp]:[hostaddr]:hostport-[guestaddr]:guestport` (protocol defaults to tcp) | | `-n` | N | Number of additional VirtIO console sockets. Creates N host socket files (`s--p<1..N>.sock` in CWD) which map to guest `/dev/ttyVI01`..`ttyVIN` (device nodes are created by basicrc, not by startnb.sh) | | `-w` | path | 9P host directory to share with guest | | `-e` | k=v,… | Export variables via QEMU fw_cfg (`opt/org.smolbsd.var.*`) | | `-E` | f=path,… | Export files via QEMU fw_cfg (`opt/org.smolbsd.file.*`) | | `-P` | (none) | Use PTY console; `startnb.sh` attaches `picocom -q -b 115200` by default. If `use_pty` in the config file is set to a command (not `y`), that command is used instead | | `-d` | (none) | Daemonize QEMU | | `-b` | (none) | Bridge networking (tap interface) | | `-N` | (none) | Disable networking | | `-s` | (none) | Share image read-write (don't lock) | | `-t` | port | TCP serial port (telnet) | | `-a` | params | Append kernel boot parameters | | `-x` | args | Extra raw QEMU arguments | | `-v` | (none) | Verbose (print QEMU command, don't execute) | | `-u` | (none) | Non-colorful output (sets `CHOUPI=`; the ASCII fallback branch in choupi is `CHOUPI=n`) | | `-h` | (none) | Show usage | **Environment variables:** - `QEMU_ACCEL` — force a specific accelerator (`kvm`, `hvf`, `nvmm`, `tcg`) - `QEMU` — override QEMU binary (default: `qemu-system-`) - `ARCH` — override architecture detection **Architecture-specific QEMU invocation:** | Arch | Machine | CPU | Accelerator | Default Kernel | |------|---------|-----|-------------|----------------| | x86_64 | `-M microvm,rtc=on,acpi=off,pic=off` | `host,+invtsc` | kvm/nvmm/hvf | `kernels/netbsd-SMOL` | | i386 | `-M microvm,…` | `host,+invtsc` | kvm/nvmm | `kernels/netbsd-SMOL386` | | aarch64 | `-M virt,gic-version=host` (host only when KVM present, else `gic-version=3`) | `host` | kvm/hvf | `kernels/netbsd-SMOL-aarch64.img` | CPU overrides: `QEMU_ACCEL=tcg` forces `qemu64` (x86) or `max` (aarch64) instead of `host`; on macOS Intel, `cputype` is always forced to `qemu64`. **Console detection:** - Checks kernel symbols for `viocon_earlyinit` via `nm` - If found: uses VirtIO console (`virtio-serial-device` + `virtconsole`) - If not: falls back to ISA serial console (`-serial stdio` or `-serial pty`) **Port forwarding format transformation:** ``` # User input: ::8080-:80 # Transformed: hostfwd=tcp::8080-:80 ``` The `-p` flag is processed by `sed` into QEMU `hostfwd=` syntax. The protocol prefix (`tcp:`, `udp:`) is optional and defaults to `tcp`. Multiple comma-separated pairs in one `-p` are all transformed. **PTY mode:** When `-P` is passed: 1. QEMU starts with `-daemonize` and writes PTY path to `qemu-.pty` 2. `startnb.sh` waits for the file, extracts the PTY path, launches `picocom -q -b 115200` (or the custom `use_pty` command) 3. On console exit, kills the QEMU process 4. This is the attach path used for services whose `CMD` runs `login -f` — the PTY console is what makes `login`'s terminal handling behave correctly; see §20.3 for a related failure mode when the image has been minimized **RTC:** Always `-rtc base=utc,clock=host,driftfix=slew` (UTC, host clock, slew driftfix) — added unconditionally. **QEMU 9.0/9.1 workaround:** Adds `-L bios -bios bios-microvm.bin` to avoid stack smashing. ### 7.2 PVH Boot Flow ``` QEMU loads netbsd-SMOL kernel directly (no BIOS, no bootloader) → Kernel initializes VirtIO MMIO devices → Kernel mounts root filesystem (ld0 / dk0 / md0) → Kernel executes /etc/rc → . /etc/include/basicrc → Sets PATH, umask, HOME → Checks for md0 (ramdisk root), remounts rw if needed → mount -a (reads /etc/fstab) → Sources /etc/rc.pre (if exists) → Loads qemufwcfg variables (if mount_qemufwcfg available) → Creates /dev/MAKEDEV, mounts ptyfs, creates fd and ttyVI* devices → Configures vioif0: 10.0.2.15/24, gateway 10.0.2.2, DNS 10.0.2.3 → Configures lo0 (IPv4 + IPv6) → Tunes TCP sendbuf/recvbuf → Sources /etc/include/mount9p (if vio9 device present) → Handles SSL_CERT_FILE (custom certs + certctl rehash) → Sources /etc/rc.local → If MOUNTRO: remount / read-only → . /etc/include/mount9p (if not already in basicrc) → Service-specific commands → CMD/ENTRYPOINT → . /etc/include/shutdown → sync, sync → If viocon: remount ro, echo 'JEMATA!' > /dev/ttyVI01 → Else: umount -af → halt -lq ``` ### 7.3 VM Control Socket When `-n N` is used (`N >= 1`): - Creates N VirtIO console sockets on the host (`s--p<1..N>.sock` in the CWD) - `/dev/ttyVI01` on the guest is the first socket (p1) - `startnb.sh` spawns a background process that monitors the p1 socket via `socat` - When guest writes `JEMATA!` to `/dev/ttyVI01`, the host kills QEMU - Guest's `shutdown` script uses this for clean host-side teardown --- ## 8. Common Runtime Scripts — Reference > Snippets below reflect the current code at the version of this doc — they are reference, not specification. If you edit files in `service/common/`, keep §8 in sync. ### 8.1 basicrc (`/etc/include/basicrc` in VM) ```sh export HOME=/ export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/pkg/bin:/usr/pkg/sbin:/rescue umask 022 # Handle ramdisk root (md0) — remount r/w if [ "$(sysctl -n kern.root_device)" = "md0" ]; then mount -u -o rw /dev/md0a / sed -i'' 's,^[^ ]*,/dev/md0a,' /etc/fstab fi mount -a # Optional pre-boot hook [ -f /etc/rc.pre ] && . /etc/rc.pre # QEMU fw_cfg variables (if mount_qemufwcfg binary exists) if [ -f /sbin/mount_qemufwcfg ]; then dmesg | grep -q qemufwcfg && . /etc/include/qemufwcfg fi # Device nodes [ ! -f "/dev/MAKEDEV" ] && cp -f /etc/MAKEDEV* /dev if [ -f /sbin/mount_ptyfs ]; then mount -t ptyfs ptyfs /dev/pts cd /dev && sh MAKEDEV fd && cd - /etc/rc.d/ttys start fi chmod 666 /dev/null # VirtIO console extra ports dmesg | grep -q viocon && \ (cd /dev && sh MAKEDEV ttyVI01 ttyVI02 && chmod 666 /dev/ttyVI*) # Static IP (faster than DHCP) if ifconfig vioif0 >/dev/null 2>&1; then route flush -inet6 >/dev/null 2>&1 ifconfig vioif0 10.0.2.15/24 route add default 10.0.2.2 mount | grep read-only || echo "nameserver 10.0.2.3" > /etc/resolv.conf fi ifconfig lo0 127.0.0.1 up ifconfig lo0 inet6 ::1 prefixlen 127 up # TCP tuning sysctl -w net.inet.tcp.sendbuf_max=16777216 sysctl -w net.inet.tcp.recvbuf_max=16777216 sysctl -w kern.sbmax=16777216 # 9P mount (if vio9 device present) . /etc/include/mount9p # Custom SSL certificates if [ -n "$SSL_CERT_FILE" ]; then [ -d "$SSL_CERT_FILE" ] && cp -R "$SSL_CERT_FILE"/* /usr/share/certs/mozilla/server || \ cp "$SSL_CERT_FILE" /usr/share/certs/mozilla/server certctl rehash 2>/dev/null fi # Optional post-boot hook [ -f /etc/rc.local ] && . /etc/rc.local # Read-only root if MOUNTRO env var set [ -n "$MOUNTRO" ] && mount -u -o ro / ``` ### 8.2 shutdown (`/etc/include/shutdown`) ```sh sync; sync dmesg | grep -q 'viocon0: adding port' && \ ( mount -u -o ro / sync; sync echo 'JEMATA!' > /dev/ttyVI01 ) || \ umount -af # no viocon(4) support halt -lq ``` The `JEMATA!` signal tells the host-side socket monitor to kill QEMU cleanly. ### 8.3 mount9p (`/etc/include/mount9p`) ```sh [ -z "$MOUNT9P" ] && MOUNT9P=/mnt if ! mount | grep -q ${MOUNT9P} && dmesg | grep -q vio9; then [ -f /etc/MAKEDEV ] && cp /etc/MAKEDEV /dev cd /dev && sh MAKEDEV vio9p0 cd - mount_9p -cu /dev/vio9p0 $MOUNT9P fi ``` ### 8.4 qemufwcfg (`/etc/include/qemufwcfg`) ```sh QEMUFWCFG=/var/qemufwcfg /sbin/mount_qemufwcfg $QEMUFWCFG for file in ${QEMUFWCFG}/opt/org.smolbsd.var.*; do [ ! -f $file ] && continue VARNAME=${file##*.} eval "export $VARNAME=\$(cat \$file)" done ``` ### 8.5 funcs (`service/common/funcs`) ```sh rsynclite() { src=$1; dst=$2 # Handle --exclude= flags # If src is a file: cp -f # If src is a directory: tar-based sync (faster than rsync for small trees) } ``` ### 8.6 vars (`service/common/vars`) ```sh BASEPATH=/mnt # Project root path (used by builder VM) DRIVE2=/drive2 # Secondary disk mount point (used by builder VM) CHOUPI=y # Enable emoji output ``` ### 8.7 choupi (`service/common/choupi`) Sets emoji or ASCII fallback icons based on `$CHOUPI`: - `CHOUPI=y` (default): emojis (➡️, ✅, ⚠️, ❌, …) - `CHOUPI=n`: ASCII fallbacks (>, \, !, X, …) Also defines `LIGHTGREEN`, `BOLD`, `NORMAL` ANSI color codes. ### 8.8 pkgin (`service/common/pkgin`) Bootstraps the pkgin package manager from scratch: 1. Detects NetBSD version → derives pkgsrc version 2. Determines architecture URL 3. Installs `pkg_install`, `pkgin`, `pkg_tarup`, `rsync`, `curl` via `pkg_add` 4. Configures repository URL 5. Installs packages from `/packages/` directory if present 6. Installs `$INSTALL_PACKAGES` if set --- ## 9. OCI Registry (Push/Pull with oras) Default registry: `ghcr.io/netbsdfr/smolbsd` Override with `SMOLREPO` environment variable. ```bash # Push ./smoler.sh push myapp-amd64:latest # → oras push ghcr.io/netbsdfr/smolbsd/myapp-amd64:latest \ # --artifact-type application/vnd.smolbsd.image \ # images/myapp-amd64.img:application/x-raw-disk-image # Pull ./smoler.sh pull myapp-amd64:latest # → oras pull ghcr.io/netbsdfr/smolbsd/myapp-amd64:latest # Downloads myapp-amd64.img to the CURRENT directory — move it to # images/ before building on top of or running it # List images with signature verification ./smoler.sh images # all images ./smoler.sh images ok # only images with valid signatures ``` `smoler/img.sh` auto-installs `oras` binary to `bin/oras` if missing. Push/pull only work on Linux and macOS (other OSes are rejected). `images` additionally: reads the `smolsig` from the last 56 bytes of each image, verifies it against the `.sig` file (uuid only), and auto-recreates a missing `.sig` file for signed images (e.g. freshly downloaded ones). **Image naming:** `-[:].img` - Tag defaults to `latest` - Signature: `smolsig:DD/MM/YYYY|UUID` appended to image and stored in `.sig` file --- ## 10. GitHub Actions CI/CD Two workflows in `.github/workflows/`: ### main.yml — builder + rescue images **Triggers:** - Push to `main` (ignoring `**.md`, `www/`, `app/`, `.github/workflows/smoler.yml`, `smolerfiles/*`) - Manual `workflow_dispatch` with inputs: `img`, `arch`, `service`, `mountro`, `curlsh` **Steps:** 1. Checkout on `ubuntu-latest` in privileged `debian:latest` container 2. Install prerequisites: `curl xz-utils make sudo git libarchive-tools rsync bmake e2fsprogs gdisk` 3. Build for both `amd64` and `evbarm-aarch64`: ```bash bmake SERVICE= CURLSH= ARCH=$arch MOUNTRO=y bmake SERVICE=rescue ARCH=$arch base # always build rescue ``` 4. Compress all `.img` files with `xz -T0 -9e` + generate SHA256 sums 5. Upload to GitHub Release tag `latest` (pre-release) via `softprops/action-gh-release@v2` ### smoler.yml — SMOLerfile service images Triggers on push to `smolerfiles/*` (or manual dispatch with a list of files). Builds a fixed set of services (`crush`, `clawd`, `bsdshell`, `nbakery`, `tiny`, `clawlite`, `ttyd`) with `smoler.sh build` on plain runners using `QEMU_ACCEL=tcg` (no KVM), after `bmake fetchimg` for amd64 and evbarm-aarch64; publishes to GitHub Packages (`packages: write`). New SMOLerfiles you want CI-tested must be added to the `DFILES` regex in this workflow. **Note:** the CI runner is Linux-only (ext2 builder path), so any fix that's specific to the NetBSD FFS builder path (WAPBL, `resize_ffs`, sailor) will not be exercised by CI — those must be tested manually on a NetBSD host. See §20. --- ## 11. BIOS Boot & Bare Metal When standard PVH boot is unavailable (Bhyve, bare metal, other VMMs): ```bash # Build with BIOS boot ./smoler.sh build -y -t USB BIOSBOOT=y BIOSCONSOLE=pc smolerfiles/Dockerfile.bsdshell # Bare metal: dd to USB drive sudo dd if=images/bsdshell-amd64:USB.img of=/dev/sde bs=1M # Bhyve/other VMMs: also SMOLIFY the kernel ./smoler.sh build -y -t freebsd BIOSBOOT=y SMOLIFY=y smolerfiles/Dockerfile.bsdshell ``` **BIOS boot internals (in mkimg.sh):** - Copies `/usr/mdec/boot` to image - Creates `/boot.cfg` with `timeout=0` and `consdev=${BIOSCONSOLE}` - After `umount`: `gpt biosboot -i 1 ${imgdev}`, `installboot /dev/r${mountdev} /usr/mdec/bootxx_ffsv1` **In kernfetch (Makefile):** - If `BIOSBOOT=y`, downloads `netbsd-GENERIC` kernel - Copies as `kernels/netbsd-GENERIC.SMOL` - If `SMOLIFY=y` and `confkerndev` exists: runs confkerndev to keep only essential drivers: `mainbus cpu acpicpu ioapic pci isa pcdisplay wsdisplay com virtio ld vioif qemufwcfg` ### confkerndev — Kernel Driver Slimming A C tool that modifies the kernel ELF binary to **disable device drivers without recompilation**: ```bash # List all drivers ./confkern …(truncated)