Linux › Section F · How user space starts › Topic F.2
systemd units and unit files
systemd describes everything it manages, from daemons to mounted filesystems, as a unit defined in a short text file. Learn to read and write those files and you can understand, change and extend how a modern Linux system starts.
Everything is a unit
systemd is goal-oriented. Each thing it can manage is a unit, and starting one is called activating it. The suffix of a unit's name gives its type. The commonest are .service units for daemons and one-off jobs, .target units that group other units into a system state, .socket units for listening sockets, .mount and .automount units for filesystems, .swap for swap space, and .timer units for scheduled tasks (see "Recurring tasks with cron and timer units"). Others include .path units that react to file changes, .device units for kernel devices reported by udev, .slice units for resource control and .scope units for processes started outside systemd. Units depend on one another, and those dependencies form a graph rather than a simple list.
Where unit files live
Unit files sit in several directories, searched in priority order. Packages install theirs in /usr/lib/systemd/system (on older Debian-family systems, /lib/systemd/system). The administrator's own units and overrides go in /etc/systemd/system, which takes precedence. Runtime units created during the current boot live under /run/systemd/system, between the two. A file in /etc/systemd/system with the same name as a packaged one replaces it completely. The rule is simple: never edit files under /usr/lib, because the next package update overwrites them; make every local change under /etc. The command systemd-analyze unit-paths lists every directory systemd searches, highest priority first.
The anatomy of a unit file
A unit file is plain text in an INI-like format derived from the XDG Desktop Entry specification. Section names appear in square brackets, each followed by Key=Value lines called directives, and lines starting with # or ; are comments. The [Unit] section is common to all types and holds a Description=, optional Documentation= links and the dependency directives. A type-specific section comes next: [Service] for a service, [Socket], [Mount] or [Timer] for those types; target units need none. An optional [Install] section is read only when the unit is enabled; the topic "systemd operation, dependencies, and targets" explains enabling. Directive names are case-sensitive: ExecStart= works, execstart= does not.
Worked example: a service for a small web app
Save this as /etc/systemd/system/inventory.service. Line 1: [Unit]. Line 2: Description=Inventory web app. Line 3: Wants=network-online.target. Line 4: After=network-online.target. Line 5: [Service]. Line 6: Type=exec. Line 7: User=inventory. Line 8: ExecStart=/usr/bin/python3 -m http.server 8080 --directory /srv/inventory. Line 9: Restart=on-failure. Line 10: RestartSec=5s. Line 11: [Install]. Line 12: WantedBy=multi-user.target. Reading it: 1. Lines 3 and 4 ask for the network to be up first. 2. Line 7 runs the server as the unprivileged user inventory, not root. 3. ExecStart= gives the command as an absolute path. 4. Lines 9 and 10 restart it 5 seconds after a crash. 5. Line 12 lets it start at boot once enabled.
Service types and readiness
systemd must know when a service counts as started, so that units ordered after it can begin. Type= tells it. Type=simple, the default when ExecStart= is set, treats the service as started as soon as the process is forked. Type=exec waits until the program has actually been executed, so a missing binary makes the start fail visibly. Type=forking suits old daemons whose first process forks a child and exits; PIDFile= helps systemd find the survivor. Type=oneshot is for a job that runs and finishes; the unit is up only when the process exits, and RemainAfterExit=yes keeps it marked active afterwards. Type=notify waits for the program to send READY=1 through sd_notify, and Type=dbus waits for it to take a D-Bus name.
Commands, variables and the environment
ExecStart= is not a shell command line. systemd splits it into words itself, so pipes, redirection with > and background & do not work; if you need them, run a script. Before the main command, ExecStartPre= can run checks, and ExecReload= defines what systemctl reload does. Environment=PORT=8080 sets a variable directly, and EnvironmentFile= reads assignments from a file; a leading minus, as in EnvironmentFile=-/etc/default/inventory, makes the file optional. In command lines, ${PORT} expands to exactly one argument, while $PORT on its own is split at whitespace. systemd also supplies variables of its own, such as $MAINPID, the tracked main process, used in ExecReload=/bin/kill -HUP $MAINPID.
Template units and specifiers
A specifier is a percent code that systemd replaces when it loads a unit: %n is the full unit name, %N the name without its suffix, %p the prefix before the @, %i the instance after it, %H the host name and %u the user running the service manager. Specifiers make template units possible. A template's name ends in @ just before the suffix, such as getty@.service. Starting getty@tty2.service makes systemd load the template and substitute tty2 wherever %i or %I appears (%I undoes systemd's escaping of special characters). One file can therefore produce many independent services, one per console, disk or website.
Worked example: one template, many instances
You back up several websites with a script that takes the site name. 1. Create /etc/systemd/system/backup@.service with [Unit], Description=Back up site %i, then [Service], Type=oneshot and ExecStart=/usr/local/bin/backup-site %i. 2. Run systemctl daemon-reload so systemd notices the new file. 3. Run systemctl start backup@shop.service. systemd loads backup@.service and replaces %i with shop, so it runs /usr/local/bin/backup-site shop. 4. For that instance, %n is backup@shop.service, %N is backup@shop, %p is backup and %i is shop. 5. Check: systemctl start backup@blog.service runs the same file with blog, and each instance has its own status and log.
Drop-ins: changing a unit without editing it
To adjust a packaged unit, add a drop-in instead of copying the whole file. For a unit foo.service, any .conf files in a directory named foo.service.d are merged on top of the main file in alphanumeric order. The command systemctl edit nginx.service opens an editor on /etc/systemd/system/nginx.service.d/override.conf and reloads systemd's configuration when you save. Write only the section headers and directives you are changing, such as [Service] and Restart=always. Most settings simply override the original, but list-type directives such as ExecStart= add to it, so to replace ExecStart= first write an empty ExecStart= line to reset it. systemctl cat nginx.service shows the main file and every drop-in, and systemctl revert nginx.service removes your changes.
Common mistakes to avoid
First, editing a file in /usr/lib/systemd/system; the next update silently undoes it. Second, putting shell syntax such as pipes or > redirection in ExecStart=, which systemd passes as literal arguments. Third, forgetting systemctl daemon-reload after creating or changing a unit file by hand, so systemd keeps using the old version. Fourth, a drop-in that sets ExecStart= without first resetting it with an empty ExecStart= line, which leaves two start commands. Fifth, choosing Type=forking for a program that stays in the foreground, so systemd waits for an exit that never comes and the start times out. Sixth, miscapitalised directives, which systemd ignores with a warning.
Key takeaways
systemd manages units, named by type: .service, .target, .socket, .mount, .timer, .path, .device, .slice and others. Packaged unit files live in /usr/lib/systemd/system, local ones in /etc/systemd/system, which wins. Unit files use [Unit], a type section such as [Service], and an optional [Install] section of case-sensitive directives. Type= (simple, exec, forking, oneshot, notify, dbus) tells systemd when a service is ready. ExecStart= is not a shell; use Environment= and EnvironmentFile= for settings. Specifiers such as %n, %p and %i let a template unit like getty@.service serve many instances. Change packaged units with drop-ins via systemctl edit, inspect them with systemctl cat, and run daemon-reload after manual edits.
Flashcards
Flashcard
Anything systemd manages, defined by a unit file
Test your understanding
Question 1 of 5
An administrator wants to change one setting of the packaged sshd.service so that the change survives updates. What should they do?
Question 2 of 5
A unit contains ExecStart=/usr/bin/myapp > /var/log/myapp.log. What happens?
Question 3 of 5
Which service type makes systemd treat a job as started only after its process has exited?
Question 4 of 5
The template backup@.service contains ExecStart=/usr/local/bin/backup-site %i. What runs for systemctl start backup@blog.service?
Question 5 of 5
A drop-in for web.service contains [Service] and the single line ExecStart=/usr/local/bin/web --fast. The service now fails to load. What is the most likely fix?
