Linux › Section H · Processes and resource utilization › Topic H.3
Tracing program execution and system calls
When a program dies at start-up or hangs without a useful message, ps and lsof have little to show. strace and ltrace let you watch every request the program makes, and the answer is usually in plain sight.
Why trace a program?
Some failures are too quick to inspect: a command exits in a fraction of a second with "error: cannot start", leaving nothing for top or lsof to see. Others are too quiet: a process sits in state S for an hour and nobody knows what it is waiting for. In both cases the program's conversation with the kernel tells the story, because nothing a program does to files, the network or other processes happens without a system call. The topic "The kernel: processes, memory, drivers, and system calls" introduced system calls as the doorway from user space into the kernel. This topic is about watching that doorway.
System calls, the C library and errno
Most programs do not issue system calls directly. They call functions in the C library, glibc on most distributions or musl on Alpine, which wrap the calls: printf() formats text in user space and eventually calls write(), and fopen() calls openat(). Library functions such as strlen() never enter the kernel at all. When a system call fails it returns an error, and the C library returns -1 and sets errno to a code naming the reason. Codes you will meet constantly are ENOENT (no such file or directory), EACCES (permission denied), EPERM (operation not permitted), EAGAIN (try again later) and EINTR (interrupted by a signal). perror() and strerror() turn a code into a readable message.
Running a command under strace
strace runs a program and prints one line per system call: the call's name, its arguments and, after an equals sign, its return value. strace cat /etc/hostname starts with execve("/usr/bin/cat", ...) = 0, then dozens of lines in which the dynamic loader opens and maps shared libraries, and finally the real work. An illustrative tail looks like this: openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3, then read(3, "web01\n", 131072) = 6, write(1, "web01\n", 6) = 6, read(3, "", 131072) = 0, close(3) = 0 and exit_group(0). The 3 is a new file descriptor, and a read that returns 0 means end of file.
Options that tame the output
strace writes to standard error, so its lines mix with the program's own errors. -o trace.txt saves them to a file instead. -e trace=openat,read limits the trace to the named calls, and classes such as -e trace=%file or -e trace=%network select whole groups. -f follows child processes created with fork or clone, and -ff with -o writes one file per process, named trace.txt.PID. -p PID attaches to a process that is already running; press Ctrl+C to detach without killing it. -tt adds a timestamp to each line, -T shows the time spent inside each call, and -s 200 prints up to 200 characters of each string instead of the default 32.
Worked example: a missing configuration file
A tool called reportd exits at once with "cannot load settings". 1. Trace only file opens: strace -e trace=openat reportd. 2. Skip the library lines and read the end (illustrative): openat(AT_FDCWD, "/etc/reportd.conf", O_RDONLY) = -1 ENOENT followed by openat(AT_FDCWD, "/home/sam/.config/reportd/settings.conf", O_RDONLY) = -1 ENOENT, then exit_group(1). 3. Interpret: the program looked in two places, found neither and gave up. 4. Check where the file really is: ls /etc/reportd/ shows reportd.conf one directory deeper, so it was installed in the wrong place. 5. Fix the location (or point the tool at it with its documented option), run again and confirm that the openat now returns a descriptor such as 3.
Worked example: what is a hung process waiting for?
A backup client, PID 2210, has shown no progress for twenty minutes. 1. Attach: sudo strace -p 2210. 2. Read the output (illustrative): the only line is read(5, and nothing more, meaning the process is blocked inside a read on descriptor 5. 3. Identify descriptor 5: sudo ls -l /proc/2210/fd/5 shows socket:[48211]. 4. Identify the socket: sudo lsof -a -p 2210 -d 5 shows a TCP connection to the backup server's port in state ESTABLISHED. 5. Conclude: the client is waiting for the server to reply, so the next step is to check the server, not the client. 6. Press Ctrl+C; strace detaches and the backup process keeps running.
Counting calls with strace -c
Sometimes you want a summary rather than every line. strace -c ls /usr prints, when the program exits, a table with one row per system call: the percentage of traced time it took, the total seconds, microseconds per call, the number of calls and the number of errors. A program that makes 40,000 tiny write() calls where a few large ones would do, or one that fails the same openat() thousands of times, stands out at once. Add -f to include child processes. Remember that a few calls never appear: time functions such as clock_gettime() usually run through the vDSO, a small piece of kernel-supplied code mapped into each process, without entering the kernel.
ltrace: watching library calls
ltrace works like strace but shows calls into shared libraries, such as malloc(), strlen(), getenv() or a function in libssl, instead of system calls. It is useful when the problem lies in how a program uses a library, for example reading an unexpected environment variable. There are far more library calls than system calls, so filter with -e, as in ltrace -e getenv somecommand. ltrace cannot see inside statically linked programs, because they contain no shared-library calls to intercept, and it is not packaged for every architecture. For production systems, perf trace and eBPF tools such as bpftrace give similar views with much lower overhead.
Permissions and overhead
strace uses the kernel's ptrace facility, which also lets debuggers control a process. You can always trace programs you start yourself. Attaching to a running process with -p needs permission: on distributions that set the Yama setting kernel.yama.ptrace_scope to 1, Ubuntu among them, an ordinary user may attach only to their own descendant processes, so attaching to anything else needs sudo. Other users' processes always need root. Tracing is also slow: every system call stops the program twice so strace can inspect it, and a busy server can run many times slower. Trace briefly, filter with -e, and avoid tracing critical production services at peak times.
Common mistakes to avoid
First, redirecting with 2> and losing track of which lines came from the program; use -o instead. Second, reading only the first lines, which are the dynamic loader opening libraries; the interesting failures are usually near the end. Third, treating every ENOENT as the problem: programs often probe several paths on purpose, so look for the last failure before the exit. Fourth, forgetting -f, so the trace stops at the parent while the child does the work. Fifth, expecting ltrace to work on a statically linked binary.
Key takeaways
Every action a program takes on files, networks or processes goes through a system call, usually via glibc or musl wrappers. Failures return -1 and set errno, such as ENOENT or EACCES. strace prints each call, its arguments and its result; -o, -e trace=, -f, -ff, -p, -tt, -T, -s and -c control it. Look near the end of a trace for the decisive failure, and use /proc/PID/fd and lsof to identify descriptors. ltrace shows shared-library calls. ptrace permissions and heavy overhead mean tracing should be brief and deliberate.
Flashcards
Flashcard
A request from a user-space program to the kernel
Test your understanding
Question 1 of 5
A trace ends with openat(AT_FDCWD, "/etc/app.conf", O_RDONLY) = -1 ENOENT, then exit_group(1). What is the most likely problem?
Question 2 of 5
Which command saves a trace of a daemon and all its child processes into one file per process?
Question 3 of 5
strace -p 2210 shows only read(5, and nothing more. What does that tell you?
Question 4 of 5
Why does a trace of a program often show no clock_gettime() calls, even though it reads the time constantly?
Question 5 of 5
On Ubuntu, an ordinary user runs strace -p on one of their own processes, started earlier from a different terminal, and is refused. Why?
