Linux › Section G · System configuration › Topic G.1

System logging: the journal and syslog

When a service fails at 3 a.m., the log is the only witness. Linux records messages from the kernel, services and login programs in two overlapping systems, the systemd journal and traditional syslog, and knowing how to query, route and trim both is a core administrator skill.

Two logging systems

Every part of a running system reports what it is doing: the kernel, daemons, login programs and your own scripts. Two mechanisms collect those messages. The first is systemd-journald, which stores entries in an indexed binary journal that you read with journalctl. The second is a syslog daemon, most often rsyslog, which writes plain text files under /var/log. Depending on the distribution you may find the journal alone, text files alone, or both side by side. Learn both, because troubleshooting often starts with whichever one the machine happens to have.

Reading the journal with journalctl

Run journalctl with no options and it pages through every stored entry, oldest first. Useful switches: -e jumps to the end; -n 20 shows the last 20 entries; -r reverses the order so the newest entry comes first; -f follows new entries as they arrive, like tail -f; -k shows only kernel messages. Each boot has its own numbering: -b means the current boot, -b -1 the one before, and journalctl --list-boots prints a table of boots with their first and last timestamps. Reading the whole journal usually needs membership of the systemd-journal or adm group, or sudo.

Filtering: unit, priority, time and text

Filters combine, and each one narrows the result. -u nginx.service keeps one unit. -p err keeps priority err and anything more severe. -S and -U (long forms --since and --until) bound the time, for example --since '2026-11-06 08:00' or --since '2 hours ago'. -g 'timeout' searches message text with a regular expression, case-insensitively when the pattern is all lowercase. -o json or -o cat changes the output format. You can also match raw journal fields, such as journalctl _SYSTEMD_UNIT=sshd.service; different fields are ANDed together, while repeating the same field ORs the values.

Worked example: why did the service fail?

The unit report.service stopped working after this morning's reboot. 1. Get a summary: systemctl status report.service shows the state and the last few log lines. 2. Pull that boot's errors: journalctl -u report.service -b -p err. 3. Read the context around the first error: journalctl -u report.service -b --since '08:00' -n 40. 4. Illustrative output might include a line such as: report[812]: cannot open /var/lib/report/data.db: Permission denied. 5. The message names the cause, a file permission problem, so check ownership and mode of that path. Each step moved from broad to narrow, and the unit and boot filters kept unrelated noise out.

Priorities from 0 to 7

Every message carries a severity: 0 emerg, 1 alert, 2 crit, 3 err, 4 warning, 5 notice, 6 info, 7 debug. Lower numbers are more serious. A single level in -p includes everything more severe, so -p warning shows levels 0 to 4, five levels in all, while -p debug shows everything. A range uses two dots, so -p err..crit selects only levels 3 and 2. Do not read -p 4 as 'level 4 only'; that is the common surprise. The same eight names appear in syslog rules, so learning them once pays twice.

Keeping the journal across reboots

The journald Storage= setting accepts volatile, persistent, auto and none. With auto, journald stores logs in /var/log/journal if that directory exists and otherwise in /run/log/journal, which is memory-backed and vanishes at reboot. auto was the built-in default until systemd 259 (December 2025), which switched to persistent; many distributions still ship older versions, so check before you need old logs. To be certain, set Storage=persistent in /etc/systemd/journald.conf or a drop-in file under /etc/systemd/journald.conf.d/, then restart systemd-journald. By default the persistent journal may use up to 10% of its filesystem, capped at 4 GiB (SystemMaxUse=). Check usage with journalctl --disk-usage; trim archived files with --vacuum-size=500M or --vacuum-time=2weeks. Vacuuming deletes entries permanently, so be sure first.

Syslog and rsyslog selectors

In the traditional rsyslog format a rule is selector, then action. A selector is facility.priority. Facilities name the sender: kern, user, mail, daemon, auth, authpriv, syslog, cron, lpr, news, uucp and local0 to local7. The priority uses the same eight names and means that level and above, so mail.warning matches mail messages of levels 0 to 4. Join selectors with a semicolon, list facilities with commas, and use none to exclude. The action is usually a file, but @host forwards over UDP and @@host over TCP; a central log server keeps a copy that survives if one machine crashes or is compromised. Configuration lives in /etc/rsyslog.conf and /etc/rsyslog.d/*.conf. rsyslogd -N1 checks the syntax before you restart the service. Test a rule with logger -p local0.notice 'test message'.

Worked example: reading two rsyslog rules

Decode these lines. Rule A: *.info;mail.none;authpriv.none /var/log/messages. 1. *.info means any facility at levels 0 to 6. 2. mail.none removes the mail facility. 3. authpriv.none removes private authentication messages, which should not sit in a widely readable file. 4. So the file holds general messages at info or worse, except mail and authpriv. Rule B: write all cron messages, any priority, to /var/log/cron.log. 5. The selector is cron.*, so the line is cron.* /var/log/cron.log. 6. Check: * as the priority matches levels 0 to 7, so even debug lines are kept. Where rsyslog is installed, Debian-family systems usually collect general messages in /var/log/syslog and logins in /var/log/auth.log; Fedora and RHEL use /var/log/messages and /var/log/secure.

Log rotation with logrotate

Text logs grow without limit, so logrotate renames, compresses and deletes them on a schedule, usually from a daily timer or cron job. It reads /etc/logrotate.conf and per-application files in /etc/logrotate.d/. A stanza might read: weekly, rotate 4, compress, missingok, notifempty inside braces after a log path. With weekly and rotate 4, you keep the active log plus four older copies, so about five weeks of history. logrotate -d /etc/logrotate.conf runs in debug mode and only reports what it would do, which makes it a safe way to test a new stanza.

Common mistakes to avoid

First, assuming the journal survives a reboot: under Storage=auto without /var/log/journal it does not. Second, reading -p warning as 'only warnings'; it includes err, crit, alert and emerg as well. Third, forgetting that a log you cannot find may be named for the distribution: syslog versus messages, auth.log versus secure. Fourth, mixing up @ (UDP, messages can be lost silently) and @@ (TCP). Fifth, running journalctl --vacuum-* or deleting logs before copying out evidence of a problem. Sixth, editing rsyslog rules without running rsyslogd -N1 first. Seventh, letting scripts log passwords or tokens, which then sit in every copy of the log.

Key takeaways

The journal is binary and queried with journalctl; syslog daemons such as rsyslog write text in /var/log. Filter with -u, -b, -p, -S/-U and -g. Priorities run 0 emerg to 7 debug, and a single -p level includes all more severe ones. Under Storage=auto the journal is persistent only if /var/log/journal exists; Storage=persistent (the default from systemd 259) removes the doubt. Syslog rules are selector then action, facility.priority, with none to exclude and @ or @@ to forward. logrotate keeps logs bounded; test it with logrotate -d. File names differ between Debian-family and Fedora/RHEL systems.

Flashcards

Flashcard

journalctl

Command that reads the systemd journal

Binary store, queried with filters.

Test your understanding

Question 1 of 5

Which command shows only error-or-worse entries from sshd.service during the previous boot?

Question 2 of 5

What does journalctl -p warning display?

Question 3 of 5

A server's journal is empty after every reboot. /var/log/journal does not exist and journald.conf sets Storage=auto. What explains this?

Question 4 of 5

What does the rsyslog selector *.info;mail.none;authpriv.none match?

Question 5 of 5

A logrotate stanza says weekly and rotate 4. How many files of that log exist at most once it is running steadily?

← Previous topicNext topic →