Containers Internals

Containers internals

Contract

Inputs

• Question or symptom (required): an OOM kill, CPU throttling, a permission denial inside the container, a mount failure, or a concept.
• Target (optional): a running container's PID, or a rootfs directory for a manual container.
• Privilege (required to know): root, or an unprivileged user relying on user namespaces. Several steps differ.

Procedure

1. Read and enter namespaces. Each namespace type isolates one resource. Done when: the target process's namespace inodes are listed and, when needed, a shell runs inside them.

ls -la /proc/self/ns/            # one link per namespace type
readlink /proc/1234/ns/net       # compare inodes to see who shares a namespace
nsenter -t <pid> -m -u -i -n -p bash
unshare --fork --mount-proc --pid --net --uts --ipc bash   # a manual container shell

#define _GNU_SOURCE
#include <sched.h>
#include <sys/mount.h>
#include <unistd.h>

static int container_init(void *arg)
{
    sethostname("container", 9);
    mount("proc", "/proc", "proc", 0, NULL);
    execv("/bin/sh", (char *[]){"/bin/sh", NULL});
    return 1;
}

static char stack[1024 * 1024];
/* The child stack grows down, so pass the top of the buffer. */
clone(container_init, stack + sizeof stack,
      CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | SIGCHLD, NULL);

2. Apply a cgroups v2 limit. The unified hierarchy at /sys/fs/cgroup exposes one directory per group; a process joins by writing its pid to cgroup.procs. Done when: the limit file holds the value and memory.events or cpu.stat shows the effect under load.

mkdir /sys/fs/cgroup/mycontainer
echo $$ > /sys/fs/cgroup/mycontainer/cgroup.procs
echo 256M > /sys/fs/cgroup/mycontainer/memory.max     # hard memory limit
echo 50 > /sys/fs/cgroup/mycontainer/cpu.weight        # share relative to siblings; default 100
echo "50000 100000" > /sys/fs/cgroup/mycontainer/cpu.max   # quota and period in microseconds
echo "default 100" > /sys/fs/cgroup/mycontainer/io.weight
cat /proc/self/cgroup
cat /sys/fs/cgroup/mycontainer/memory.events           # oom and oom_kill counters

3. Build the filesystem with overlayfs. Reads fall through to the lower layers; writes land in the upper directory; workdir is overlayfs bookkeeping and must be empty and on the same filesystem as upperdir. Done when: the merged mount shows the union and a write appears only in upperdir.

mount -t overlay overlay -o lowerdir=lower1:lower2,upperdir=upper,workdir=work merged

Docker's overlay2 driver keeps its layers under /var/lib/docker/overlay2.

4. Run the bundle with runc. runc spec writes a config.json whose ociVersion matches the installed runtime (runc 1.5.1 writes 1.3.0); do not hand-edit that field. Edit process.args, linux.namespaces, linux.resources, process.capabilities, and linux.seccomp. Done when: runc run starts the container and runc list shows it.

mkdir -p mycontainer/rootfs        # populate rootfs first
runc spec -b mycontainer           # add --rootless when not root
runc run -b mycontainer mycontainer
runc list

5. Filter syscalls with seccomp-bpf. Done when: the filter loads and the blocked syscall returns the chosen action.

#include <seccomp.h>
#include <errno.h>

scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ALLOW);
seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(mount), 0);
seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(pivot_root), 0);
seccomp_load(ctx);

Docker's default profile ships in the moby repository under profiles/seccomp/default.json. To find the syscall a profile is missing, run the workload under strace -f first.

6. Drop capabilities. A container process should run as non-root with the smallest bounding set that still works. Done when: capsh --print or getcap shows only the intended capabilities.

capsh --drop=all --caps="cap_net_bind_service+eip" -- -c '/app/server'
setcap cap_net_bind_service+ep /usr/bin/myserver
getcap /usr/bin/myserver

7. Explain rootless mode. A user namespace maps container uid 0 to an unprivileged host uid, so root inside is an ordinary user outside. Rootless containers cannot mount most filesystems and hold no CAP_SYS_ADMIN on the host. Done when: /proc/<pid>/uid_map for the container shows the mapping.

cat /proc/self/uid_map      # "0 1000 1" maps container uid 0 to host uid 1000

8. Stack the escape mitigations: user namespace, seccomp, a mandatory access control profile (AppArmor or SELinux), a dropped capability set, a read-only root, no-new-privileges, and Landlock for filesystem scope. Known escape vectors are a mounted container-engine socket, privileged mode, kernel CVEs, and /proc leaks. Done when: each layer in use is named and the missing ones are listed.

docker run --read-only --cap-drop=ALL --security-opt=no-new-privileges \
  --security-opt seccomp=default.json myimage

For policy depth (SELinux, AppArmor, seccomp) use kernel-security. To trace what a container does at the syscall level, use ebpf. For the kernel side of cgroups and namespaces, use kernel-internals.

Failure and recovery

Output

The primitive used for each isolation or limit, the commands run, the observed effect, and the teardown list: cgroup directories to remove, mounts to unmount, and processes to stop.