Linux › Section F · How user space starts › Topic F.7
Emergency booting and single-user mode
Sooner or later a machine will refuse to boot normally, or you will be locked out of it. Knowing the rescue options, from a minimal systemd target to a live USB stick, turns a crisis into a routine repair.
A ladder of recovery options
When a system will not start properly, work up from the least drastic fix. 1. If a new kernel is the problem, choose an older kernel from the GRUB menu. 2. Boot to a minimal systemd state, rescue.target or emergency.target, by adding a parameter to the kernel command line. 3. Stop inside the initial RAM filesystem, or replace init with a shell, when even those fail. 4. Boot a separate live or rescue image from USB and repair the installed system from outside. Each step gives you less of the normal system but depends less on it working. Before changing anything, read the evidence: journalctl -xb shows the current boot's log with explanations, and journalctl -b -1 the previous boot's.
Single-user mode, then and now
Single-user mode boots quickly to a root shell without the usual crowd of services. Under System V init it was runlevel 1, as "System V runlevels and init" described. Under systemd there are two levels. rescue.target pulls in the base system, mounts local filesystems and starts a root shell, but no network or login services. emergency.target is far more minimal: it starts a shell on the console with the root filesystem usually still mounted read-only and almost nothing else running. Both ask for the root password before giving you the shell, and typing exit, or running systemctl default, continues to the normal default target. The price of either mode is few amenities: no network and no graphical desktop.
Getting there from the boot menu
To enter a rescue state for one boot, interrupt GRUB, highlight the entry and press e to edit it. Find the line that begins with linux, which holds the kernel command line, and add a parameter at its end: systemd.unit=rescue.target, or its shorter forms rescue, single or 1, for rescue mode; systemd.unit=emergency.target, or emergency, for emergency mode. Press Ctrl+X or F10 to boot. The edit is not saved, so the next boot is normal again. On a running system, systemctl rescue and systemctl emergency switch to the same targets, but they stop most services and cut off remote users, so use them only at the console.
When there is no root password
The rescue and emergency shells ask for the root password through a program called sulogin. Some distributions, Ubuntu among them, lock the root account by default, and then sulogin refuses access. There are two ways round this. Adding init=/bin/bash to the kernel line makes a shell PID 1 with no password at all; the root is usually mounted read-only, so mount -o remount,rw / is needed before any change. Nothing else is running, so after the repair run sync and then exec /sbin/init to continue booting. Or boot a live image instead. Note what this means for security: anyone at the console can do the same, unless the boot loader has a password and the disk is encrypted.
Worked example: a bad /etc/fstab entry
A data disk was removed, but its line in /etc/fstab remains, and the boot drops to emergency mode. 1. Enter the root password at the prompt. 2. Run journalctl -xb and look for the failed mount: a timeout waiting for the device behind /data. 3. Run mount -o remount,rw / so the root can be edited. 4. Edit /etc/fstab: comment out the /data line, or add the nofail option so a missing disk no longer stops the boot, as "Mounting filesystems and /etc/fstab" explains. 5. Run systemctl daemon-reload, because systemd builds mount units from fstab. 6. Run mount -a; if it reports no errors, the remaining entries mount correctly. 7. Run systemctl default to finish booting. Check: the next reboot completes.
Worked example: resetting a lost root password
On Fedora and Red Hat systems, which use dracut, the documented method stops inside the initramfs. 1. At the GRUB menu press e, add rd.break to the linux line and press Ctrl+X. 2. At the switch_root prompt, the real root is mounted read-only at /sysroot; run mount -o remount,rw /sysroot. 3. Run chroot /sysroot so commands act on the installed system. 4. Run passwd root and enter the new password twice. 5. Run touch /.autorelabel; changing the password outside SELinux leaves /etc/shadow wrongly labelled, and this file makes the next boot relabel the filesystem. 6. Type exit twice to leave the chroot and continue booting. Check: the relabelling boot is slow, then the new password works. Some releases still ask for a password here; a live image then works instead.
Live and rescue images
A live image is a complete Linux system that boots and runs from removable media without being installed; most distributions' installation images double as live systems, and dedicated rescue images such as SystemRescue carry extra repair tools. Booted this way, the installed system is just a set of disks you can examine safely, with networking and a full set of tools available. Typical jobs are checking filesystems after a crash, repairing /etc/fstab or /etc/passwd, resetting a password, reinstalling a boot loader and restoring files from backups. Because nothing on the damaged system needs to work, this is nearly always the most comfortable rescue method.
Repairing from a live image
1. Identify the installed system's partitions with lsblk and blkid; device names can differ from the installed system's, so check sizes and labels. If the root is encrypted, unlock it first with cryptsetup open, and activate any LVM volumes with vgchange -ay. 2. Check an unmounted filesystem with fsck if needed. Warning: never run fsck on a mounted filesystem. 3. Mount the root, for example mount /dev/nvme0n1p2 /mnt. 4. Bind the live system's kernel filesystems into it: for d in dev proc sys run; do mount --bind /$d /mnt/$d; done. 5. Run chroot /mnt, make the repair, such as rebuilding the initramfs, and exit. 6. Run umount -R /mnt to unmount everything before rebooting.
Common mistakes to avoid
First, reinstalling at once instead of reading journalctl -xb, which usually names the failing unit. Second, editing files in emergency mode without remounting the root read-write, then wondering why changes fail. Third, running fsck on a mounted filesystem, which can corrupt it. Fourth, using systemctl rescue over SSH, which cuts off your own connection. Fifth, forgetting touch /.autorelabel after a password reset on an SELinux system, so the new password is refused. Sixth, mounting the wrong partition from a live image because device names differ; confirm with lsblk and blkid. Seventh, leaving init=/bin/bash in the saved GRUB configuration rather than as a one-time edit.
Key takeaways
Recover from the least drastic option upwards: an older kernel, rescue or emergency mode, an initramfs or init=/bin/bash shell, then a live image. rescue.target gives a root shell with local filesystems mounted; emergency.target gives a bare shell, usually with a read-only root. Enter them for one boot by pressing e in GRUB and adding systemd.unit=rescue.target or systemd.unit=emergency.target (or rescue, single, 1, emergency). Both need the root password; if root is locked, use init=/bin/bash or live media. Remount the root read-write before editing. On dracut systems, rd.break plus chroot /sysroot and touch /.autorelabel resets a root password. From a live image, bind-mount dev, proc, sys and run, then chroot.
Flashcards
Flashcard
Root shell with the base system and local filesystems, no network
Test your understanding
Question 1 of 5
Which kernel parameter, added in the GRUB editor, boots a systemd machine into rescue mode for one boot?
Question 2 of 5
In emergency mode you try to fix /etc/fstab, but the editor cannot save the file. What should you run first?
Question 3 of 5
An Ubuntu system's root account is locked, and rescue mode refuses access. Which option still gives a root shell?
Question 4 of 5
After resetting the root password through rd.break on a Fedora system, the new password is rejected. What step was most likely missed?
Question 5 of 5
You are repairing an installed system from a live USB. Which practice is correct?
