inspektor-gadget Linux host / container-runtime troubleshooting with Inspektor Gadget
The standalone ig binary runs the same eBPF gadgets as the Kubernetes
integration, but against a single Linux host and its local container runtime
(Docker / containerd / CRI-O / podman) — no cluster, no kubectl. Use it on
VMs, edge nodes, CI runners, or any box where you can run sudo ig. Events are
enriched with container name + runtime (not pod/namespace) when a runtime is
present; with --host you also see host (non-container) processes.
Reach for ig when host logs/metrics can't answer "what is the kernel doing
right now?" — a syscall failing silently, a connection reset before the app
notices, a file open returning ENOENT.
Prerequisite: confirm IG is available (check first, then route)
This skill drives the standalone ig binary. Confirm it's usable before the loop:
command -v ig >/dev/null 2>&1 && ig version || echo "ig MISSING -> references/install.md"
If ig is missing, open references/install.md and follow it — don't guess
an install command. ig needs root (CAP_BPF / CAP_SYS_ADMIN) and a BTF-enabled
kernel; ask the operator before installing on a host you don't own.
The one rule: discover, don't guess
Never hardcode a gadget's flags or field names from memory. Enumerate the real interface at run time:
sudo ig run <gadget>:latest --help # flags + a "--fields" block listing every data source & field
sudo ig run <gadget>:latest --timeout 5 -o json \
| jq -s '(.[0] // {}) | if type == "array" then (.[0] // {}) else . end | keys'
Each gadget is an OCI image pulled on demand; there is no list-gadgets
command and no fixed list to memorize. The gadget's own --help/--fields is
the source of truth. See references/discovering-params-and-fields.md.
The loop (repeat until root cause)
- Route. Map the symptom to a domain (networking / security /
process-lifecycle / storage-fs / performance) and a candidate gadget using
the table below +
references/gadget-catalog.md. - Discover. Run
<gadget>:latest --helpto read the real flags and fields. - Run bounded. Scope and time-box:
sudo ig run <gadget>:latest --runtimes containerd -c <name> --timeout <sec> -o json(-cfilters by container,--comm/--pidby process; add--hostfor host processes; always set--timeout;--max-entriesfortop/snapshot). Seereferences/common-flags.md. - Read the columns. Inspect the fields (
runtime.*,proc.*, error codes) to confirm or refute, then narrow and repeat.
Symptom → first gadget (confirm flags/fields with --help)
| Symptom | Domain | Start with | Then / disambiguate |
|---|---|---|---|
| DNS fails / slow / NXDOMAIN | networking | trace_dns |
latency in latency_ns |
| Connection reset / refused / hangs | networking | trace_tcp |
trace_tcpretrans, trace_tcpdrop |
| Packet loss / high latency | networking | trace_tcpdrop |
profile_tcprtt, top_tcp, tcpdump |
| TLS/cert/SNI issue | networking | trace_sni |
trace_ssl |
| Port bind fails / "address already in use" | networking | snapshot_socket |
trace_bind (bind + errno) |
| Process exits / restarts unexpectedly | process-lifecycle | trace_exec |
trace_signal (filter --signal 9/15 — Go SIGURG / glibc SIGRTMIN async-preempt noise), trace_oomkill |
| Process killed / OOM | process-lifecycle | trace_oomkill |
top_process, profile_cpu |
| Container died too fast to trace live (post-mortem) | process-lifecycle | traceloop |
replays recent syscalls from the ring buffer; empty if it wasn't already recording |
| Watch an interactive shell / tty / pts / keystrokes | process-lifecycle | ttysnoop |
no --tty/--pts selector — scope by --pid/--comm/container |
| "no such file" / missing path | storage-fs | trace_open --failed |
snapshot_file, top_file |
| Slow disk / file I/O | storage-fs | trace_fsslower |
profile_blockio, top_blockio |
| Unexpected mount/umount, or suspicious hardlink/symlink (escape) | storage-fs | trace_mount |
trace_link (type = HARDLINK/SYMLINK cuts the noise) |
| fd leak / "too many open files" / inotify watch storm | storage-fs | fdpass |
fsnotify; snapshot_file (unclosed fds in one proc) |
| Permission denied despite correct FS perms | security | trace_capabilities |
audit_seccomp; host audit logs for AppArmor/SELinux |
| Seccomp denial | security | audit_seccomp |
the code + syscall identify the seccomp action; advise_seccomp to author a profile |
| AppArmor/SELinux denial | security | host audit logs | trace_lsm can correlate hook activity, but does not expose another LSM's verdict |
| Kernel module loaded / rootkit / unexpected insmod | security | trace_init_module |
trace_capabilities (CAP_SYS_MODULE) |
| Harden / author a seccomp or NetworkPolicy profile | security | advise_seccomp |
advise_networkpolicy |
| High CPU, or slow despite low CPU% (CFS throttling / cgroup CPU limit) | performance | profile_cpu |
top_cpu_throttle (capped?), top_process |
| Memory growth / leak (userspace) | performance | trace_malloc (libc malloc only — statically-linked Go runtime allocator invisible) |
trace_malloc --collect-ustack (the leak site) |
| App hung / mutex deadlock | performance | deadlock (pthread only — Go sync.Mutex invisible) |
profile_cpu |
| GPU / CUDA out-of-memory | performance | top_cuda_memory |
device vs pinned; profile_cuda = libcuda Driver-API |
| Quantify eBPF/gadget CPU or memory overhead (self-profiling) | meta | bpfstats |
needs an active window — longer --timeout |
This is a deliberately thin shortlist, not the catalog — it names the few
gadgets that fit the most common symptoms so you don't scan the full bundled set,
keeping this always-loaded router small. Symptom not listed, or
unsure which row fits? Read references/gadget-catalog.md — it groups all
bundled gadgets by domain with the disambiguation reasoning (which of the
TCP/file/TLS/CPU gadgets to pick). The authoritative live list is whatever sudo ig run <name>:latest --help resolves; confirm a gadget's flags/fields there
before relying on them.
References (load only the one you need)
references/gadget-catalog.md— all upstream gadgets grouped by domain, with per-domain disambiguation.references/discovering-params-and-fields.md— discover-don't-guess mechanics (igvariant).references/common-flags.md— host/container scope, output, timeouts, gotchas.references/install.md— detect whetherigis usable; install it + requirements (root, BTF) if missing.references/kubernetes-companion.md— when to switch tokubectl gadget(clusters).
Per-domain deep-dive playbooks (shared with the k8s skill)
The five per-domain playbooks (networking, security, process-lifecycle,
storage-fs, performance) live in the kubernetes-troubleshooting skill's
references/ tree and apply verbatim here — the gadgets, flags, and fields are
identical; only the launcher and scope flags differ. To use one from this
single-host skill, swap kubectl gadget run→sudo ig run and -n/-p→
-c/--host (full mapping in references/kubernetes-companion.md):
../kubernetes-troubleshooting/references/domain-networking.md— DNS / TCP / drops / retransmits / TLS-SNI / pcap / NetworkPolicy.../kubernetes-troubleshooting/references/domain-security.md— capabilities / LSM / seccomp / kernel-module loading.../kubernetes-troubleshooting/references/domain-process-lifecycle.md— exec / signals / OOM / snapshots / traceloop / tty.../kubernetes-troubleshooting/references/domain-storage-fs.md— open / slow FS / block I/O / mounts / links / fd / fsnotify.../kubernetes-troubleshooting/references/domain-performance.md— CPU / throttle / RTT / deadlock / malloc / GPU.
Safety
Observation is read-only — gadgets never modify the host or containers. ig
needs elevated privileges (typically sudo, CAP_BPF/CAP_SYS_ADMIN) to load eBPF.
Always bound streaming gadgets with --timeout and cap top/snapshot with
--max-entries.