Linux › Section H · Processes and resource utilization › Topic H.4
Threads
Many of the programs that matter most on a server, from web servers to databases, are one process running many threads at once. Seeing those threads, and understanding what they share, explains odd figures such as a process using 380% of the CPU.
One process, several flows of execution
A thread is a single flow of execution: a position in the program, a set of CPU register values and a stack for local variables. Every process starts with one thread, the main thread, and a process with only that thread is single-threaded. The main thread may create more, making the process multithreaded. The kernel schedules each thread separately, so on a machine with several cores the threads of one process can run at the same moment. The topic "Coordinating the machine: processes and scheduling" in the Computer Science course gives the general theory; here the focus is on how Linux implements threads and how you see them.
What threads share and what they keep
The threads of one process share almost everything that belongs to the process: its virtual memory (code, global variables and the heap), its open file descriptors, its current directory, its user and group IDs and its signal handlers. Each thread keeps its own stack, its own register values (including the instruction pointer), its own signal mask and its own thread ID. Separate processes are the opposite: after fork(), parent and child have separate memory, so a change made by one is invisible to the other. That difference is the whole trade-off: sharing makes threads fast to create and cheap to communicate with, but it also lets them damage each other's data.
Why programs use threads
There are two main reasons. The first is parallel computation: a video encoder or a compiler can split work across every core, finishing several times faster than with one thread. The second is waiting: a server handling hundreds of connections can give each a thread that blocks on its own network reads while the others carry on. Both jobs could be done with several processes, and some servers do exactly that, but creating a thread is cheaper than creating a process, and threads exchange data simply by reading and writing shared memory instead of sending it through a pipe or socket.
How Linux implements threads
Linux has no separate kind of object for threads. Every thread is a task, created with the clone() system call, and the flags passed to clone() decide how much the new task shares with its creator. A new process shares little; a new thread shares memory, descriptors and signal handlers. Each task has its own ID, the TID. The tasks that make up one process form a thread group, whose ID is the TID of the main thread; this group ID is what ps and kill call the PID. So the main thread's TID equals the PID, and the other threads have TIDs of their own. In /proc/PID/status the Tgid line gives the group ID and Threads the count.
Viewing threads with ps and top
By default ps and top show one line per process. To see threads: ps -L -p PID adds an LWP column (light-weight process, another name for the TID) with one line per thread, and ps -eLf lists every thread on the system, with an NLWP column counting threads per process. ps -o pid,nlwp,comm shows just the counts. ps -T gives a similar list with an SPID column, and the BSD-style ps m prints each thread beneath its process. In top, press H, or start it with top -H, to switch to one line per thread; the PID column then shows TIDs. A STAT value containing l, such as Sl, marks a multithreaded process.
Worked example: counting a server's threads
A server written in Go, running in a test container, is examined (illustrative output, trimmed; PIDs will differ). 1. Find the PID: pgrep -o ollama returned 9. 2. Count its threads: ps -o pid,nlwp,comm -p 9 printed 9 13 ollama, so 13 threads. 3. List them: ps -L -o pid,lwp,stat,comm -p 9 showed rows 9 9 Sl, 9 11 Sl, 9 12 Sl and so on. Every row has PID 9; the LWP column gives each thread's TID, and the first, 9, is the main thread. 4. Read the state: Sl means sleeping and multithreaded. 5. Cross-check with the kernel: grep Threads /proc/9/status printed Threads: 13, and ls /proc/9/task would list the same 13 TIDs.
Threads in resource monitoring
Because threads run at the same time, their CPU use adds up within one process. In top, %CPU is measured against one core, so a process whose four busy threads each use about 95% of a core shows 4 × 95 = 380%. On an 8-core machine the ceiling is 800%. When a process looks unusually busy, press H in top to see whether one thread is spinning or all of them are working. Threads also count against limits: the per-user process limit (ulimit -u) and a cgroup's pids.max both count threads, so a program that creates thousands of threads can fail with "Resource temporarily unavailable" even though the system has few processes.
Programming threads: pthreads and locks
Most Linux programs create threads through POSIX threads (pthreads), provided by the C library: pthread_create() starts a thread running a given function, and pthread_join() waits for it to finish. Languages such as Go, Java, Rust and Python sit on top of the same kernel tasks. Because threads share memory, any data changed by more than one thread must be protected. A mutex (pthread_mutex_lock() and pthread_mutex_unlock()) lets only one thread at a time run the protected code, and condition variables let a thread sleep until another signals that something has changed. On Linux these are built on futexes, which avoid entering the kernel when no other thread holds the lock.
Worked example: a lost update
Two threads each add 1 to a shared counter one million times, with no lock. 1. Note that counter = counter + 1 is three steps: read the value, add 1, write it back. 2. Suppose the counter holds 41. Thread A reads 41, and before A writes, thread B also reads 41. 3. A writes 42; B, which also computed 41 + 1, writes 42 too. 4. Two increments happened, but the counter rose by only one: an update was lost. 5. Repeated across two million increments, the final total usually falls short of 2,000,000, and by a different amount on each run. 6. The fix: hold a mutex around the read-add-write so that each update completes before the next begins. This bug is called a data race.
Common mistakes to avoid
First, expecting ps or top to show threads by default; use ps -L, ps -eLf or top's H key. Second, reading a %CPU above 100 as an error; it is the sum of several threads. Third, thinking threads have private global variables; only stacks and registers are private, so shared data needs a lock. Fourth, trying to stop one thread with kill: a signal sent to a PID is delivered to the whole process, and a fatal signal ends every thread. Fifth, forgetting that thread counts use up the same process limits.
Key takeaways
A thread is a flow of execution with its own stack and registers that shares its process's memory, descriptors and IDs. Linux implements threads as tasks made with clone(); a process is a thread group whose PID is the main thread's TID. ps -L, ps -eLf, ps -T, ps m, top -H and /proc/PID/task show threads, and NLWP or the Threads line counts them. %CPU can exceed 100 for multithreaded processes, and threads count against ulimit -u and pids.max. pthreads, mutexes and futexes let programs share data safely; without them, data races lose updates.
Flashcards
Flashcard
A flow of execution inside a process
Test your understanding
Question 1 of 5
Which of these do all threads of one process share?
Question 2 of 5
ps -L -p 9 shows rows with LWP values 9, 11 and 12, all with PID 9. What is LWP 9?
Question 3 of 5
On a 4-core machine, top shows one process at 310% CPU. What does this mean?
Question 4 of 5
Two threads each increment a shared counter 500,000 times without a lock. What is the most likely result?
Question 5 of 5
A program fails to create new threads with "Resource temporarily unavailable", although ps shows few processes. What should you check?
