Linux › Section H · Processes and resource utilization › Topic H.1
Tracking processes
A single ps listing is a photograph; troubleshooting usually needs a film. This topic shows how to watch processes change over time with top, how to read their family tree and states, and how to ask the kernel directly through /proc.
From a snapshot to a live view
The topic "Listing and manipulating processes: signals and jobs" used ps to list what is running at one instant. That finds a PID, but it cannot show which process keeps eating the CPU or whether memory use is growing. top redraws its display every few seconds (3 by default in the procps-ng version found on most distributions), with the busiest processes at the top. Alongside it, this topic uses three other sources: the parent-child tree, the one-letter process state, and the /proc filesystem, which is where ps and top get their numbers in the first place.
Driving top from the keyboard
top is interactive, and its single-key commands are case-sensitive. P sorts by current CPU use (the default), M by resident memory, and T by total CPU time. u shows only one user's processes, c toggles the full command line, V shows a tree view, f chooses columns, H shows one line per thread, 1 splits the CPU summary per core and q quits. From the command line, top -p 1234 watches only PID 1234, and top -b -n 1 prints one batch-mode snapshot that can be saved or piped to grep.
Reading the top header
The first five lines summarise the whole system. Line 1 shows the time, the uptime, the number of logged-in users and three load averages, explained in the topic "Resource monitoring: CPU, memory, and I/O". Line 2 counts tasks by state, including zombies. Line 3 splits CPU time into us (user code), sy (kernel), ni (niced), id (idle), wa (I/O wait), hi and si (interrupts) and st (stolen by a hypervisor). Lines 4 and 5 give memory and swap. In the table, VIRT is virtual size, RES the memory actually in RAM, S the state and TIME+ the CPU time used.
Process families: PPID and the tree
Every process except the first records the PID of the process that created it, its PPID. Because each parent was created by another, processes form a tree with PID 1, normally systemd, at the root. The tree shows, for example, which service a runaway worker belongs to. ps -ef --forest draws branches with \_ markers, ps -o pid,ppid,stat,cmd shows the family columns for the processes of your current terminal, and pstree prints a compact drawing (it comes in the psmisc package, which some minimal systems lack). pgrep -a sshd prints the PID and full command line of every process whose name matches, which is quicker than scrolling.
Process states
The STAT column of ps and the S column of top show a one-letter state. R means running or ready to run. S is interruptible sleep: waiting for input, a timer or a network reply, which is where most processes spend most of their time. D is uninterruptible sleep, usually waiting for disk or network storage; a process in D usually cannot be killed until the I/O finishes. T means stopped, for example by Ctrl+Z or a debugger. Z is a zombie, explained next. ps adds modifiers after the letter: s marks a session leader, + a process in the foreground group of its terminal, l a multithreaded process, and < or N a raised or lowered priority.
Zombies and orphans
When a process exits, the kernel keeps a tiny entry holding its exit status until the parent collects it with a wait call. Until then the dead process is a zombie, shown as Z and <defunct>. A zombie uses no CPU and almost no memory, and kill cannot remove it, because it has already exited. The fix is to deal with the parent: a well-written parent reaps its children, and if the parent itself exits, the zombie is adopted by PID 1 (or a designated subreaper process), which reaps it. An orphan is the opposite case: a live process whose parent has exited. It, too, is adopted, and its PPID changes to 1 or to the subreaper's PID.
Worked example: finding a zombie's parent
This illustrative output comes from a test in which a parent deliberately never waits for its child (PIDs will differ on your system). 1. List families with ps -o pid,ppid,stat,cmd, which showed: 7 1 S sleep 60 and 9 7 Z [sleep] <defunct>. 2. Read the zombie's line: PID 9 is in state Z. 3. Read its PPID column: 7, so the parent is PID 7, the sleep 60 process, which never calls wait. 4. Confirm from the kernel: grep -E '^(State|PPid)' /proc/9/status printed State: Z (zombie) and PPid: 7. 5. Decide: killing PID 9 does nothing; ending or fixing PID 7 lets PID 1 adopt and reap the zombie.
/proc: the kernel's view of each process
/proc is a virtual filesystem: its files are generated by the kernel when read and take no disk space. Each process has a directory named after its PID, and /proc/self always means the process doing the reading. Useful entries include status (name, state, PPID, UIDs, thread count, memory such as VmRSS), cmdline (the arguments, separated by NUL bytes), environ (the environment, readable only by the owner or root), cwd and exe (symbolic links to the working directory and the program file), fd (one link per open file descriptor) and task (one subdirectory per thread). Because cmdline uses NUL separators, tr '\0' ' ' < /proc/1/cmdline makes it readable.
Worked example: reading /proc for one process
Suppose top shows PID 4210 using a lot of memory and you want to know exactly what it is (illustrative output). 1. Find the program: ls -l /proc/4210/exe shows a link to /usr/bin/python3. 2. Find the arguments: tr '\0' ' ' < /proc/4210/cmdline prints python3 /opt/report/build.py --full. 3. Find where it runs: ls -l /proc/4210/cwd points to /opt/report. 4. Check its family and size: grep -E '^(PPid|Threads|VmRSS)' /proc/4210/status shows PPid: 1, Threads: 1 and VmRSS: 812340 kB, about 793 MiB. 5. Conclude: a report script started by the service manager is using about 0.8 GiB of RAM.
htop and atop
Two optional tools build on top. htop adds per-core meters, scrolling, a tree view (F5), searching (F3) and filtering (F4), and it can send signals (F9) or change niceness (F7 and F8) without typing PIDs. atop records snapshots at intervals into log files, so you can look back at what was running when a problem happened, which ordinary top cannot do. Install them with sudo apt install htop on Debian and Ubuntu or sudo dnf install htop on Fedora and RHEL. All of them read the same /proc data as top.
Common mistakes to avoid
First, trying to kill a zombie: it is already dead, so look at its parent instead. Second, treating a state of S as a problem; sleeping is normal, while many processes stuck in D point to storage trouble. Third, forgetting that top's keys are case-sensitive: lowercase m changes the memory display, while capital M sorts by memory. Fourth, reading VIRT as memory in use; RES is the part actually in RAM. Fifth, mistaking a permission error on another user's /proc/PID/environ for an empty environment.
Key takeaways
top gives a refreshing view of processes, sorted with P, M and T, filtered with u, and printed once with top -b -n 1. Processes form a tree rooted at PID 1; ps -ef --forest, pstree and pgrep -a show families. States are R, S, D, T and Z, with modifiers such as s, + and l. A zombie is an exited process awaiting its parent's wait call; fix the parent, not the zombie. Orphans are adopted by PID 1 or a subreaper. /proc/PID holds status, cmdline, environ, cwd, exe, fd and task, all generated by the kernel on demand.
Flashcards
Flashcard
Live, refreshing view of processes, busiest first
Test your understanding
Question 1 of 5
ps shows a process in state Z marked <defunct>. What is the right way to get rid of it?
Question 2 of 5
In top, which key sorts the process list by resident memory use?
Question 3 of 5
Which file shows a process's state, its parent's PID and its thread count in readable text?
Question 4 of 5
Many processes show state D, the load average is high and the CPUs are mostly idle. What does state D suggest?
Question 5 of 5
A shell exits while a background job it started is still running. What happens to that job's PPID if it survives?
