Strace Ltrace

strace and ltrace

strace (6.19 on the grounding host) prints every system call a process makes with its arguments, return value, and errno. ltrace (0.7.91) does the same for calls that go through PLT slots into shared libraries. Together they answer "what did the program ask the system for, and what did it get back" without a debugger or a rebuild, which is the fastest route to a missing file, a refused connection, or a permission error.

Contract

Inputs

• The command line to trace, or the PID of a running process for -p.
• The symptom, which picks the filter: file not found (%file), network (%network), process spawn (%process), memory (%memory), signals (%signal), descriptors (%desc), or everything (all).
• ptrace permission: attaching to another user's process or a process in a container needs CAP_SYS_PTRACE (containers: --cap-add=SYS_PTRACE); /proc/sys/kernel/yama/ptrace_scope at 1 restricts attaching to descendants.

Procedure

1. Trace the whole run to a file first: strace -f -o trace.txt ./myapp args. -f follows forks and clones and prefixes lines with the PID; -o keeps the trace out of the program's own stderr. Attach to a live process with strace -p <pid> -o trace.txt. Done when: trace.txt ends with +++ exited with N +++ or the attach is confirmed.

2. Filter to the syscall class the symptom points at. The class names take a % prefix (file without the prefix still works but is deprecated):

   strace -e trace=%file ./myapp          # open, openat, stat, access, unlink, rename
   strace -e trace=%network ./myapp       # socket, connect, bind, accept, sendto, recvfrom
   strace -e trace=%process ./myapp       # fork, clone, execve, wait4, exit_group
   strace -e trace=%memory ./myapp        # mmap, munmap, mprotect, brk
   strace -e trace=%signal ./myapp        # kill, rt_sigaction, rt_sigprocmask
   strace -e trace=%desc ./myapp          # close, dup, poll, epoll_wait
   strace -e trace=openat,read,write ./myapp
   

   Done when: the trace holds only the class under suspicion.

3. Show only failures. strace -Z prints only syscalls that returned an error (-z prints only successes); combine with a class: strace -Z -e trace=%file ./myapp. Each failing line has the form openat(AT_FDCWD, "/etc/myapp.conf", O_RDONLY) = -1 ENOENT (No such file or directory), which names the path the program expected. Done when: the first failing call relevant to the symptom is identified.

4. Read the errno:

   errno	Meaning	Usual cause
   ENOENT	No such file or directory	Wrong path, missing config, missing shared library
   EACCES	Permission denied	File mode, ownership, SELinux or AppArmor label
   EPERM	Operation not permitted	Missing capability, seccomp, needs setuid
   EADDRINUSE	Address already in use	Port still bound by another process
   ECONNREFUSED	Connection refused	Nothing listening on the target port
   ETIMEDOUT	Connection timed out	Route or firewall drops the packets
   EAGAIN	Resource temporarily unavailable	Non-blocking descriptor; a retry is expected
   EBADF	Bad file descriptor	Descriptor closed or never opened
   ENOMEM	Out of memory	Allocation or mmap refused
   ENOEXEC	Exec format error	Binary for another architecture or a bad interpreter line

   Done when: the errno is mapped to a cause in the program's own terms.

5. Add detail where the default output hides it. -s 256 raises the string cutoff from 32 bytes; -y appends the path behind each descriptor (3</etc/ld.so.cache>); -t, -tt, -ttt add wall-clock timestamps at increasing precision; -r prints relative timestamps; -T prints the time spent inside each call; -k prints a stack trace per syscall so the caller in the program is visible without gdb; -e verbose=all expands structures; -i prints the instruction pointer. Done when: the failing call carries the detail the diagnosis needs.

6. Profile with the summary when the symptom is slowness: strace -c ./myapp prints one row per syscall with % time, seconds, usecs/call, calls, and errors. A high usecs/call marks a blocking call; a non-zero errors column marks retries; a high call count on read or write with small sizes marks missing buffering. Done when: the top syscall by time and the top by error count are named.

7. Trace library calls with ltrace when the failure is in user space above the syscall layer. Filters use + between symbols and glob patterns: ltrace -e malloc+free+fopen ./myapp, ltrace -e 'pthread_*' ./myapp, ltrace -e 'dlopen+dlsym+dlclose' ./myapp. -l libfoo.so* traces every call into one library; -S adds syscalls; -n 2 indents nested calls; -c prints a call-count summary; -p <pid> attaches; -s 256 widens strings. Each line reads malloc(1024) = 0x55a1b2c3d000. ltrace sees only PLT calls, so a statically linked or LTO-inlined call does not appear. Done when: the library call and its return value are named.

8. Verify the diagnosis by fixing the cause (create the file, open the port, grant the permission) and rerunning the same filtered trace. Done when: the failing line no longer appears and the program behaves.

Recipes for the common cases live in references/strace-patterns.md.

Failure and recovery

Output

A diagnosis naming the failing syscall or library call with its full line from the trace, the errno and its cause in the program's terms, the fix applied or proposed, and the confirming rerun; plus the trace file paths.