Linux › Section H · Processes and resource utilization › Topic H.5

Resource monitoring: CPU, memory, and I/O

Processes compete for three hardware resources: processor time, memory and input/output. A handful of standard tools show how the kernel is sharing them out, and reading their numbers correctly is the first step in explaining why a system feels slow.

Measure before you change anything

Most Linux systems run well on their default settings, and tuning a system that is short of nothing wastes time. Take one resource at a time and ask how busy it is, whether work is queuing for it and which process is responsible. top, vmstat and free come from the procps-ng package, installed on almost every system; iostat, pidstat and sar come from sysstat, which you may need to install with apt or dnf.

CPU time: user, system and real

The time command reports how long a command took and where the time went. Bash's built-in version prints three figures. real is the elapsed wall-clock time from start to finish. user is CPU time spent running the program's own code. sys is CPU time the kernel spent working for it, such as reading files. If real is much larger than user plus sys, the program spent most of its life waiting: for the disk, for a network reply or for its turn on a busy CPU. The separate GNU program, run as /usr/bin/time -v (often a package of its own), also reports peak memory and page faults.

Scheduling and nice values

The kernel's scheduler decides which runnable thread gets each CPU. Linux used the Completely Fair Scheduler from 2007; kernel 6.6 (released in 2023) replaced it with EEVDF, which also aims for fairness while serving short, latency-sensitive tasks better. You can influence it with a nice value from -20 (most favoured) to 19 (least favoured); the default is 0. top shows it in the NI column next to PR, the priority the kernel derives from it. nice -n 10 make starts a command at nice 10, and renice -n 15 -p 4321 changes a running process. Any user may make a process nicer, but only root (strictly, a process with CAP_SYS_NICE) may lower a nice value. Real-time policies, SCHED_FIFO and SCHED_RR, give time-critical tasks strict priority.

Load averages

uptime, top and /proc/loadavg show three load averages, for roughly the last 1, 5 and 15 minutes. Each is an exponentially weighted average of the number of tasks that are running, waiting for a CPU, or in uninterruptible sleep (state D). The D part is specific to Linux: tasks waiting for disk or network storage count, even though they use no CPU. To judge a load figure, divide it by the number of CPU cores, which nproc prints. About 1.0 per core means the CPUs are just fully occupied; well above 1.0 means tasks are queuing. Comparing the three figures shows the trend: a 1-minute value above the 15-minute value means load is rising.

Worked example: reading uptime

A 2-core server prints: load average: 3.70, 1.85, 0.92. 1. Count cores: nproc prints 2. 2. Divide each figure by 2 (computed): 3.70 gives 1.85 per core, 1.85 gives about 0.93 and 0.92 gives 0.46. 3. Read the trend: the 1-minute figure is twice the 5-minute figure and four times the 15-minute one, so the load has climbed steeply in the last few minutes. 4. Judge the level: 1.85 per core means on average almost two tasks want each core, so roughly half of them are waiting. 5. Next step: run top. If %CPU is high, find the busy processes; if the CPUs are mostly idle while many tasks are in state D, the load is coming from storage waits instead.

Memory: pages, faults and free

The kernel manages memory in pages, usually 4,096 bytes (getconf PAGE_SIZE prints the size), and loads them on demand. When a process touches a page the memory management unit cannot find, a page fault occurs. A minor fault is cheap: the page is already in RAM and only needs mapping. A major fault means reading from disk or swap, and many of them slow everything. ps -o pid,min_flt,maj_flt,cmd shows the counts for a process. free -h summarises the system; its most useful column is available, the memory that can be given to programs without swapping, including cache the kernel can drop. A small free figure on its own is normal.

vmstat: the whole system over time

vmstat 2 prints a line every two seconds; the first line averages everything since boot, so read from the second line on. r counts runnable tasks and b tasks blocked on I/O; swpd, free, buff and cache are memory in KiB. si and so show memory swapped in from and out to disk per second. bi and bo show blocks read from and written to devices. The cpu columns give percentages, as in top: us, sy, id, wa (waiting for I/O) and st, and recent versions add gu for guest time. Steady non-zero so values, a falling free figure and rising wa together signal memory pressure.

The out-of-memory killer

If memory and swap both run out, the kernel cannot satisfy page requests and invokes the OOM (out-of-memory) killer. It chooses a victim by a badness score, roughly how much memory the process would free, adjusted by its oom_score_adj value, and kills it with SIGKILL. The kernel log records the victim, so check journalctl -k or dmesg when a service vanishes without an error of its own. /proc/PID/oom_score shows each process's current score.

Disk I/O: iostat, iotop and ionice

iostat -dx 2 prints an extended device report every two seconds, including the average wait per request (r_await and w_await, in milliseconds) and %util, the share of time the device was busy; near 100% with long waits means saturated. iotop, run as root, shows which threads are doing the I/O. Its PRIO column shows the I/O scheduling class: be (best effort, the normal class) with a level from 0 to 7, rt (real time) or idle. ionice -c 3 -p 4321 moves a process into the idle class, so it does I/O only when nothing else needs the disk; classes work only with an I/O scheduler that honours them, such as BFQ.

Worked example: per-process figures with pidstat

A nightly import job, PID 5120, seems to slow the whole server. 1. Watch its CPU: pidstat -u -p 5120 5 prints a line every 5 seconds; %CPU stays near 4, so CPU is not the problem. 2. Watch its memory: pidstat -r -p 5120 5 shows RSS steady and majflt/s near 0, so it is not causing heavy paging. 3. Watch its disk: pidstat -d -p 5120 5 shows kB_wr/s of about 85,000, roughly 83 MiB per second (illustrative). 4. Confirm on the device: iostat -dx 2 shows the data disk at 97% util with long w_await. 5. Act without stopping the job: ionice -c 3 -p 5120 lets other processes' I/O go first, and renice -n 10 -p 5120 lowers its CPU priority too.

Common mistakes to avoid

First, comparing a load average with 1 on a multicore machine; divide by the core count. Second, reading high load as high CPU use, when Linux also counts tasks waiting in state D. Third, panicking at a small free figure instead of checking available. Fourth, reading vmstat's first line as current activity; it is the average since boot. Fifth, expecting renice 20 to mean anything beyond 19, the lowest priority.

Key takeaways

time splits a command's life into real, user and sys; a big gap means waiting. The scheduler (EEVDF since kernel 6.6) shares CPUs, guided by nice values from -20 to 19, set with nice and renice. Load averages count running, runnable and D-state tasks over about 1, 5 and 15 minutes; divide by the core count. Memory is managed in pages; major faults hit the disk, and free's available column matters most. vmstat shows the whole system over time, the OOM killer logs its victims in the kernel log, and iostat, iotop, ionice and pidstat cover disks and single processes.

Flashcards

Flashcard

real / user / sys

Elapsed time / CPU in the program / CPU in the kernel

From the time command.

Test your understanding

Question 1 of 5

A 4-core server shows load averages of 2.0, 3.0 and 6.0 (1, 5 and 15 minutes). Which reading is correct?

Question 2 of 5

time reports real 6.20s, user 0.40s, sys 0.30s for a download script. What does this suggest?

Question 3 of 5

free -h shows 300Mi free but 5.1Gi available on an 8 GiB machine. What should you conclude?

Question 4 of 5

An ordinary user runs renice -n -5 -p 4321 on their own process. What happens?

Question 5 of 5

A database disappears overnight with no error in its own log. Where should you look first?

← Previous topicNext topic →