Linux › Section H · Processes and resource utilization › Topic H.6

Control groups and performance tuning

Watching resources is half the job; the other half is deciding how much of them each workload may use. Control groups let the kernel account for and limit whole groups of processes, and a disciplined tuning method keeps changes to the ones that actually help.

From watching to limiting

nice and ionice only change priorities; they cannot stop a runaway program from taking all the memory. The traditional per-process limits, set with ulimit in a shell or prlimit for a running process (the POSIX rlimit interface), cap things such as open files or core-dump size, but each process is limited separately, and a program that forks fifty children gets fifty allowances. A control group (cgroup) solves this. It is a kernel feature that places a set of processes in a group and applies accounting and limits to the group as a whole, through controllers: cpu, memory, io and pids, among others. cgroups belong to the kernel, but on most systems systemd creates and manages them.

Version 1 and version 2

There are two designs. In cgroup v1, each controller had its own separate hierarchy, so one process could sit in one group for cpu and a different group for memory. In cgroup v2, there is a single unified hierarchy: each process belongs to exactly one cgroup, and that cgroup can have several controllers enabled. v2 has been the default on Fedora since 2019 and on Debian 11, Ubuntu 21.10 and RHEL 9 by 2022, and systemd 258 (2025) removed support for v1 altogether. You can tell which you have from mount: a cgroup2 filesystem at /sys/fs/cgroup means v2. On a pure v2 system, cat /proc/self/cgroup prints a single line starting 0::, such as 0::/user.slice/user-1001.slice/session-4.scope for a login shell.

The cgroup filesystem

cgroups have no special system calls for everyday use: they are files and directories under /sys/fs/cgroup. Each directory is a cgroup, and its subdirectories are child cgroups. Files whose names start with cgroup. manage the group itself: cgroup.procs lists its processes (cgroup.threads lists threads), cgroup.controllers lists the controllers available to it and cgroup.subtree_control lists those passed down to its children. Controller files carry the controller's name: memory.current and memory.max, pids.current and pids.max, cpu.max, cpu.weight and cpu.stat, io.max and io.stat. A value of max means no limit at this level, although a parent cgroup further up may still impose one.

Worked example: reading a container's limits

Docker builds containers on cgroups, which makes them a handy place to see limits. A container was started with --memory=256m --cpus=0.5 --pids-limit=100 and its files were read from inside (illustrative output, matching Docker's documented behaviour). 1. cat /sys/fs/cgroup/memory.max printed 268435456. Divide by 1,024 twice: 268,435,456 bytes is exactly 256 MiB, so the m in 256m means MiB. 2. cpu.max printed 50000 100000: in each period of 100,000 microseconds the group may run for 50,000, which is half of one CPU. 3. pids.max printed 100, and pids.current printed 2, so two tasks were running. 4. cpu.weight printed 100, the default share when CPUs are contended. 5. /proc/self/cgroup showed only 0::/, because a cgroup namespace makes the container's own group look like the root.

Changing cgroups by hand

As root you can write to these files directly, though on a systemd system you normally let systemd do it. Creating a cgroup is just mkdir inside the tree; the kernel fills in the interface files. echo 4321 > cgroup.procs moves a process into a group, and echo 200 > pids.max caps the number of tasks. Three rules catch people out. A controller must be enabled in the parent's cgroup.subtree_control (for example by writing +memory +pids) before children can use it. Apart from the root, processes may live only in leaf cgroups, not in a group whose children have controllers enabled. And an empty cgroup is removed with rmdir, even though its interface files are still listed.

systemd slices, scopes and services

systemd arranges cgroups in a tree of units: system.slice holds services, user.slice holds login sessions and user services, and machine.slice holds virtual machines and containers. systemd-cgls prints that tree with the processes in each group, and systemd-cgtop shows live CPU, memory and I/O use per cgroup, like top for groups. Limits are set as unit properties: systemctl set-property nginx.service MemoryMax=1G CPUQuota=50% applies at once and persists (add --runtime for a temporary change), or you can put the same settings in a unit file's [Service] section. systemd-run --scope -p MemoryMax=500M ./build.sh runs a one-off command in a limited scope. The topic "Containers: namespaces and cgroups" builds on all of this.

Worked example: a memory limit in action

A test container was limited to 64 MiB of memory with no extra swap, and a Python command inside tried to allocate a 200 MiB buffer (illustrative output). 1. The command printed Killed and the shell reported exit=137. 2. Decode the exit status: 137 - 128 = 9, and signal 9 is SIGKILL. 3. Check the cgroup's event counters: cat /sys/fs/cgroup/memory.events showed max 37, oom 1 and oom_kill 1. 4. Interpret: the group hit memory.max 37 times, the kernel reclaimed what it could, and when that was not enough its OOM killer acted inside the cgroup only. 5. Conclude: the rest of the system was untouched; only the process in the limited group was killed, which is exactly what a memory limit is for.

A method for performance tuning

Tuning without measurement is guesswork. Brendan Gregg's USE method gives a checklist: for every resource (CPUs, memory, disks, network interfaces) check its utilisation (how busy it is), saturation (how much work is queued for it) and errors. The tools are those of the previous topic plus network counters: mpstat -P ALL for each core, vmstat for memory pressure, iostat -dx for disks, and ip -s link and ss -s for interfaces and sockets. perf top shows which functions use the CPU, and iperf3 measures the bandwidth between two hosts. Find the one resource that is saturated before changing anything, change one thing at a time, measure again and keep notes, so that every change can be reversed. Fixing inefficient work usually beats adding hardware.

Tuning tools and history

Kernel parameters are read and set with sysctl: sysctl vm.swappiness shows how readily the kernel swaps (0 to 200 on current kernels, default 60), sysctl -w vm.swappiness=10 changes it until reboot, and a line in a file such as /etc/sysctl.d/90-local.conf, loaded with sysctl --system, makes it permanent. On Red Hat-family systems, tuned applies whole profiles, such as tuned-adm profile virtual-guest. For history, sar (sysstat) records statistics so you can see what happened at 3 a.m. yesterday, acct (process accounting) logs each finished command's resource use and disk quotas cap users' storage. eBPF tools such as bpftrace trace kernel events with little overhead. Be wary of copied tuning lists: many settings are obsolete.

Common mistakes to avoid

First, relying on ulimit to contain a program that forks many children; each child gets its own allowance, while a cgroup limits the group. Second, assuming a limit of 256m means 256,000,000 bytes, when memory.max shows it as 256 MiB, 268,435,456 bytes. Third, writing limits by hand into cgroups that systemd manages; systemd may undo them, so use set-property or unit files. Fourth, expecting a child cgroup to use a controller its parent has not enabled in cgroup.subtree_control. Fifth, tuning several sysctl settings at once and never measuring which one helped.

Key takeaways

A cgroup groups processes so the kernel can account for and limit them together through controllers such as cpu, memory, io and pids. cgroup v2 uses one hierarchy, one group per process, at /sys/fs/cgroup; systemd 258 dropped v1. Key files are cgroup.procs, cgroup.controllers, cgroup.subtree_control, memory.max, cpu.max, pids.max and cpu.stat. systemd manages the tree as slices, scopes and services; use systemd-cgls, systemd-cgtop, systemctl set-property and systemd-run. Tune with the USE method, one measured change at a time, using sysctl, tuned and sar.

Flashcards

Flashcard

cgroup

Kernel group of processes with shared accounting and limits

Managed by systemd on most systems.

Test your understanding

Question 1 of 5

A cgroup's cpu.max file contains 25000 100000. How much CPU may the group use?

Question 2 of 5

Why is a cgroup better than ulimit for capping the total memory of a build that forks many compiler processes?

Question 3 of 5

A process in a cgroup with memory.max set exits with status 137, and memory.events shows oom_kill 1. What happened?

Question 4 of 5

You want nginx.service never to use more than 1 GiB of memory, surviving reboots. What is the best approach on a systemd system?

Question 5 of 5

In the USE method, what does saturation measure for a disk?

← Previous topicNext topic →