systemd Knowledge Patch
Load this skill before changing units, boot images, service sandboxes, networkd configuration, image workflows, user sessions, or systemd-facing C and shell code. Start with the project or host's actual systemd version and use only guidance that applies to it.
Reference Index
| Reference |
Topics |
| Boot, UKIs, and TPM Policy |
UKI profiles, DeviceTree and firmware selection, Secure Boot, PCR-lock, NvPCRs, early boot |
| Compatibility, Builds, and Administration |
Removed interfaces, platform requirements, runtime-loaded libraries, configuration search, OS metadata |
| Containers and Virtual Machines |
nspawn, vmspawn, capsules, unprivileged containers, per-user daemons, SSH transports, MStacks |
| Credentials and Enrollment |
User-bound credentials, null-key policy, cryptenroll, FIDO2, storage keys, certificates |
| Devices and Administrative Tools |
Udev testing, ACLs, device identity, tmpfiles purge, sysusers, factory reset |
| Homed, Users, and Sessions |
Homed records and areas, PAM classes, pidfd sessions, inhibitors, sleep, run0 |
| Images, Storage, and Extensions |
Image transfer and pulls, DDIs, repart, integrity, sysext/confext, sysupdate |
| Network Configuration |
Forwarding, bridge VLANs, DHCP, cellular links, routes, offloads, networkd Varlink |
| Resolution, Observability, and IPC |
Journal and coredumps, DNS delegation, JSON/Varlink APIs, transactions, event loops |
| Resource Control and Sandboxing |
Cgroup accounting, namespaces, BPF delegation, private filesystems, quotas, oomd |
| Units and Activation |
Mount dependencies, socket descriptors, timers, transient services, reload, readiness |
Breaking Changes and Required Migrations
Use cgroup v2 exclusively
- Remove legacy and hybrid cgroup v1 boot and build configuration.
- Do not depend on
SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1; it was transitional
and cgroup v1 support was subsequently removed.
- Account for HugeTLB memory in cgroup memory use on supporting kernels.
Replace SysV and rc.local logic
- Convert SysV scripts and
/etc/rc.local work into native service, socket, timer, path, or target units.
- Compatibility targets do not restore script loading.
- Explicitly enable each required getty instance:
systemctl enable --now getty@tty1.service
Account for changed defaults
- Journald uses persistent storage by default even without a pre-created
/var/log/journal.
- TTY and PTY nodes default to mode
0600; do not assume group-write access.
- Networkd and nspawn require nftables for NAT.
- Systemd-boot and systemd-stub TPM integration requires TPM 2.0.
- Main configuration files may be selected from
/etc, /run, /usr/local/lib, or /usr/lib, in that order.
- A drop-in ending in
.ignore remains installed but is inactive.
Repair udev ACL rules
Rules granting uaccess must survive change events and sort before rule 73:
ACTION!="remove", SUBSYSTEM=="hidraw", TAG+="uaccess"
Verify and test rules without side effects:
udevadm verify /etc/udev/rules.d/60-example.rules
udevadm test --verbose /sys/class/hidraw/hidraw0
Use exact builtin names; prefix abbreviations are rejected. Do not assign
v258 device ownership to non-system accounts without checking whether the
local release contains the later partial revert.
Treat tmpfiles purge as explicit destruction
Purge requires named configuration files and affects only entries marked $:
d$ /var/lib/example 0755 root root -
Preview precisely the same invocation first:
systemd-tmpfiles --dry-run --purge example.conf
systemd-tmpfiles --purge example.conf
Declare runtime-loaded package dependencies
ELF scanners may not discover compression, crypto, kmod, PAM, ACL, blkid, seccomp, SELinux, or most libmount integrations because they are loaded at runtime. Add explicit feature dependencies; missing libkmod can prevent boot.
systemd-analyze dlopen-metadata /usr/lib/systemd/systemd
Units and Service Sandboxing
Prefer current namespace modes
[Service]
PrivateUsers=managed
PrivatePIDs=yes
PrivateTmp=disconnected
ProtectControlGroups=strict
ProtectHostname=private:worker
PrivateBPF=yes
PrivateUsers=managed obtains a transient 65,536-ID range from nsresourced; identity maps the first 65,536 IDs and full the complete 32-bit range.
PrivateTmp=disconnected gives /tmp and /var/tmp separate tmpfs mounts.
- Delegate namespaces and BPF operations only when the workload needs them.
Use current activation and reload controls
[Service]
ExecReload=/usr/bin/example reload
ExecReloadPost=/usr/libexec/example-reload-finished
RefreshOnReload=yes
MemoryTHP=never
[Socket]
ListenStream=/run/example.sock
PassFileDescriptorsToExec=yes
PassPIDFD=yes
AcceptFileDescriptors=yes
- Validate both
LISTEN_PID and LISTEN_PIDFDID when PID identity matters.
RefreshOnReload= governs attached extensions and credentials.
- A leading
| on an Exec*= command invokes a shell; other shell syntax is not interpreted automatically.
Avoid timer stampedes
[Timer]
RandomizedOffsetSec=10min
DeferReactivation=yes
The randomized offset is stable, while deferred reactivation discards an expiration that happened while the service was still active.
Images, Boot, and TPM
Use DDI and version-pick workflows
- Put UAPI-versioned alternatives in a directory ending
.v/ and select them
with systemd-vpick.
- Use
importctl for tar, raw, filesystem, extension, portable-service, nspawn, and vmspawn image transfer.
- Use
root=dissect or mount.usr=dissect for automatic DDI and Verity discovery; automatically dissected XBOOTLDR partitions must be VFAT.
Build and update partition images carefully
- The last duplicate partition definition wins in image mount options.
AddValidateFS= records filesystem-use constraints; a mismatch requested
through x-systemd.validatefs causes an immediate reboot.
- Use
systemd-repart - to calculate minimum image size without modifying a
device.
- Repart can configure dm-integrity and volume-key pinning.
Refresh extensions intentionally
System and configuration extension refresh is a no-op when the image set did not change. Use --always-refresh=yes only for a forced unmount/remount, and use RefreshOnReload= when service reload must refresh extensions and credentials too.
Build multi-profile UKIs
- Use
.profile sections for normal, debug, and recovery profiles in one UKI.
- Use
.dtbauto, .hwids, and the system hardware-ID catalog for automatic DeviceTree selection.
- Keep offline PCR signing and Secure Boot prepare/sign/join phases distinct.
- New TPM enrollments use an empty PCR mask; add managed PCR-lock and signed
PCR 11 policy when required.
- PCR-lock excludes PCR 12 by default because a UKI credential is measured
there itself.
Networking and Resolution
Replace broad forwarding settings
[Network]
IPv4Forwarding=yes
IPv6Forwarding=yes
IPForward= is deprecated. Once [BridgeVLAN] has any valid setting, it is authoritative and undeclared VLAN IDs are removed from the interface.
Preserve dynamic configuration deliberately
- Use
KeepConfiguration=dynamic or dynamic-on-stop; DHCP-only names are
obsolete.
- A networkd restart preserves DHCPv4, DHCPv6, NDISC, and IPv4LL state.
- Mutable netdev and traffic-control changes reload in place; immutable identifiers still require interface recreation.
- Treat
DNSOverTLS=yes as strict certificate verification on affected point
releases.
Use scoped resolution and cellular links
DNS delegate files define independent domain-specific servers and routing or search domains, with optional FirewallMark=. Networkd can configure cellular links through [MobileNetwork], including APN, authentication, roaming, PIN, operator, route metric, IP family, and gateway behavior.
Observability, Sessions, and Privilege
Query one service invocation
journalctl --list-invocation -u example.service
journalctl --invocation=ID -u example.service
Use --follow --synchronize-on-exit=yes for reliable streaming shutdown. Programmatic clients can retrieve journal entries over Varlink.
Prefer public JSON and Varlink APIs
sd-json and sd-varlink are public libsystemd APIs.
- Varlink supports descriptor passing, SSH execution, a public socket
registry, and pluggable transport bridges.
- Manager APIs expose execution settings, unit filters, reload/reexecute, and
transaction diagnostics.
- Unit counters distinguish kernel
OOMKills from oomd ManagedOOMKills.
Choose PAM session classes explicitly
Lightweight background classes do not start a user manager. Set PAM class= or XDG_SESSION_CLASS when a full user manager is required; use class=none when no logind session should be created. Logind ends pidfd-tied sessions as soon as their leader exits; the old returned descriptor is not a lifetime anchor.
Use run0 and inhibitors deliberately
- Interactive
run0 defaults to late PTY attachment.
run0 --empower preserves identity and home while granting capabilities
and the Polkit empower group.
run0 --area= enters a homed area; it changes $HOME and
$XDG_RUNTIME_DIR but does not isolate files from the owning UID.
- Ordinary
block inhibitors affect root and their holder. Bypass explicitly
with --force or --check-inhibitors=no; use block-weak only when those
older semantics are intended.
1---2name: systemd-knowledge-patch3description: systemd4license: MIT5---678# systemd Knowledge Patch910Load this skill before changing units, boot images, service sandboxes, networkd configuration, image workflows, user sessions, or systemd-facing C and shell code. Start with the project or host's actual systemd version and use only guidance that applies to it.1112## Reference Index1314| Reference | Topics |15| --- | --- |16| [Boot, UKIs, and TPM Policy](references/boot-uki-and-tpm.md) | UKI profiles, DeviceTree and firmware selection, Secure Boot, PCR-lock, NvPCRs, early boot |17| [Compatibility, Builds, and Administration](references/compatibility-build-and-administration.md) | Removed interfaces, platform requirements, runtime-loaded libraries, configuration search, OS metadata |18| [Containers and Virtual Machines](references/containers-and-virtual-machines.md) | nspawn, vmspawn, capsules, unprivileged containers, per-user daemons, SSH transports, MStacks |19| [Credentials and Enrollment](references/credentials-and-enrollment.md) | User-bound credentials, null-key policy, cryptenroll, FIDO2, storage keys, certificates |20| [Devices and Administrative Tools](references/devices-and-administration.md) | Udev testing, ACLs, device identity, tmpfiles purge, sysusers, factory reset |21| [Homed, Users, and Sessions](references/homed-users-and-sessions.md) | Homed records and areas, PAM classes, pidfd sessions, inhibitors, sleep, run0 |22| [Images, Storage, and Extensions](references/images-storage-and-extensions.md) | Image transfer and pulls, DDIs, repart, integrity, sysext/confext, sysupdate |23| [Network Configuration](references/networking.md) | Forwarding, bridge VLANs, DHCP, cellular links, routes, offloads, networkd Varlink |24| [Resolution, Observability, and IPC](references/resolution-observability-and-ipc.md) | Journal and coredumps, DNS delegation, JSON/Varlink APIs, transactions, event loops |25| [Resource Control and Sandboxing](references/resource-control-and-sandboxing.md) | Cgroup accounting, namespaces, BPF delegation, private filesystems, quotas, oomd |26| [Units and Activation](references/units-and-activation.md) | Mount dependencies, socket descriptors, timers, transient services, reload, readiness |2728## Breaking Changes and Required Migrations2930### Use cgroup v2 exclusively3132- Remove legacy and hybrid cgroup v1 boot and build configuration.33- Do not depend on `SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1`; it was transitional34 and cgroup v1 support was subsequently removed.35- Account for HugeTLB memory in cgroup memory use on supporting kernels.3637### Replace SysV and rc.local logic3839- Convert SysV scripts and `/etc/rc.local` work into native service, socket, timer, path, or target units.40- Compatibility targets do not restore script loading.41- Explicitly enable each required getty instance:4243```sh44systemctl enable --now getty@tty1.service45```4647### Account for changed defaults4849- Journald uses persistent storage by default even without a pre-created50 `/var/log/journal`.51- TTY and PTY nodes default to mode `0600`; do not assume group-write access.52- Networkd and nspawn require nftables for NAT.53- Systemd-boot and systemd-stub TPM integration requires TPM 2.0.54- Main configuration files may be selected from `/etc`, `/run`, `/usr/local/lib`, or `/usr/lib`, in that order.55- A drop-in ending in `.ignore` remains installed but is inactive.5657### Repair udev ACL rules5859Rules granting `uaccess` must survive `change` events and sort before rule 73:6061```udev62ACTION!="remove", SUBSYSTEM=="hidraw", TAG+="uaccess"63```6465Verify and test rules without side effects:6667```sh68udevadm verify /etc/udev/rules.d/60-example.rules69udevadm test --verbose /sys/class/hidraw/hidraw070```7172Use exact builtin names; prefix abbreviations are rejected. Do not assign73v258 device ownership to non-system accounts without checking whether the74local release contains the later partial revert.7576### Treat tmpfiles purge as explicit destruction7778Purge requires named configuration files and affects only entries marked `$`:7980```text81d$ /var/lib/example 0755 root root -82```8384Preview precisely the same invocation first:8586```sh87systemd-tmpfiles --dry-run --purge example.conf88systemd-tmpfiles --purge example.conf89```9091### Declare runtime-loaded package dependencies9293ELF scanners may not discover compression, crypto, kmod, PAM, ACL, blkid, seccomp, SELinux, or most libmount integrations because they are loaded at runtime. Add explicit feature dependencies; missing libkmod can prevent boot.9495```sh96systemd-analyze dlopen-metadata /usr/lib/systemd/systemd97```9899## Units and Service Sandboxing100101### Prefer current namespace modes102103```ini104[Service]105PrivateUsers=managed106PrivatePIDs=yes107PrivateTmp=disconnected108ProtectControlGroups=strict109ProtectHostname=private:worker110PrivateBPF=yes111```112113- `PrivateUsers=managed` obtains a transient 65,536-ID range from nsresourced; `identity` maps the first 65,536 IDs and `full` the complete 32-bit range.114- `PrivateTmp=disconnected` gives `/tmp` and `/var/tmp` separate tmpfs mounts.115- Delegate namespaces and BPF operations only when the workload needs them.116117### Use current activation and reload controls118119```ini120[Service]121ExecReload=/usr/bin/example reload122ExecReloadPost=/usr/libexec/example-reload-finished123RefreshOnReload=yes124MemoryTHP=never125126[Socket]127ListenStream=/run/example.sock128PassFileDescriptorsToExec=yes129PassPIDFD=yes130AcceptFileDescriptors=yes131```132133- Validate both `LISTEN_PID` and `LISTEN_PIDFDID` when PID identity matters.134- `RefreshOnReload=` governs attached extensions and credentials.135- A leading `|` on an `Exec*=` command invokes a shell; other shell syntax is not interpreted automatically.136137### Avoid timer stampedes138139```ini140[Timer]141OnCalendar=hourly142RandomizedOffsetSec=10min143DeferReactivation=yes144```145146The randomized offset is stable, while deferred reactivation discards an expiration that happened while the service was still active.147148## Images, Boot, and TPM149150### Use DDI and version-pick workflows151152- Put UAPI-versioned alternatives in a directory ending `.v/` and select them153 with `systemd-vpick`.154- Use `importctl` for tar, raw, filesystem, extension, portable-service, nspawn, and vmspawn image transfer.155- Use `root=dissect` or `mount.usr=dissect` for automatic DDI and Verity discovery; automatically dissected XBOOTLDR partitions must be VFAT.156157### Build and update partition images carefully158159- The last duplicate partition definition wins in image mount options.160- `AddValidateFS=` records filesystem-use constraints; a mismatch requested161 through `x-systemd.validatefs` causes an immediate reboot.162- Use `systemd-repart -` to calculate minimum image size without modifying a163 device.164- Repart can configure dm-integrity and volume-key pinning.165166### Refresh extensions intentionally167168System and configuration extension refresh is a no-op when the image set did not change. Use `--always-refresh=yes` only for a forced unmount/remount, and use `RefreshOnReload=` when service reload must refresh extensions and credentials too.169170### Build multi-profile UKIs171172- Use `.profile` sections for normal, debug, and recovery profiles in one UKI.173- Use `.dtbauto`, `.hwids`, and the system hardware-ID catalog for automatic DeviceTree selection.174- Keep offline PCR signing and Secure Boot prepare/sign/join phases distinct.175- New TPM enrollments use an empty PCR mask; add managed PCR-lock and signed176 PCR 11 policy when required.177- PCR-lock excludes PCR 12 by default because a UKI credential is measured178 there itself.179180## Networking and Resolution181182### Replace broad forwarding settings183184```ini185[Network]186IPv4Forwarding=yes187IPv6Forwarding=yes188```189190`IPForward=` is deprecated. Once `[BridgeVLAN]` has any valid setting, it is authoritative and undeclared VLAN IDs are removed from the interface.191192### Preserve dynamic configuration deliberately193194- Use `KeepConfiguration=dynamic` or `dynamic-on-stop`; DHCP-only names are195 obsolete.196- A networkd restart preserves DHCPv4, DHCPv6, NDISC, and IPv4LL state.197- Mutable netdev and traffic-control changes reload in place; immutable identifiers still require interface recreation.198- Treat `DNSOverTLS=yes` as strict certificate verification on affected point199 releases.200201### Use scoped resolution and cellular links202203DNS delegate files define independent domain-specific servers and routing or search domains, with optional `FirewallMark=`. Networkd can configure cellular links through `[MobileNetwork]`, including APN, authentication, roaming, PIN, operator, route metric, IP family, and gateway behavior.204205## Observability, Sessions, and Privilege206207### Query one service invocation208209```sh210journalctl --list-invocation -u example.service211journalctl --invocation=ID -u example.service212```213214Use `--follow --synchronize-on-exit=yes` for reliable streaming shutdown. Programmatic clients can retrieve journal entries over Varlink.215216### Prefer public JSON and Varlink APIs217218- `sd-json` and `sd-varlink` are public libsystemd APIs.219- Varlink supports descriptor passing, SSH execution, a public socket220 registry, and pluggable transport bridges.221- Manager APIs expose execution settings, unit filters, reload/reexecute, and222 transaction diagnostics.223- Unit counters distinguish kernel `OOMKills` from oomd `ManagedOOMKills`.224225### Choose PAM session classes explicitly226227Lightweight background classes do not start a user manager. Set PAM `class=` or `XDG_SESSION_CLASS` when a full user manager is required; use `class=none` when no logind session should be created. Logind ends pidfd-tied sessions as soon as their leader exits; the old returned descriptor is not a lifetime anchor.228229### Use run0 and inhibitors deliberately230231- Interactive `run0` defaults to late PTY attachment.232- `run0 --empower` preserves identity and home while granting capabilities233 and the Polkit `empower` group.234- `run0 --area=` enters a homed area; it changes `$HOME` and235 `$XDG_RUNTIME_DIR` but does not isolate files from the owning UID.236- Ordinary `block` inhibitors affect root and their holder. Bypass explicitly237 with `--force` or `--check-inhibitors=no`; use `block-weak` only when those238 older semantics are intended.