Linux › Section J · Network applications and services › Topic J.2

Network servers and SSH

Almost every Linux server is managed over SSH, and most of its other services follow the same pattern of a daemon listening on a port. Setting up the SSH server properly, with key-based logins and sensible defaults, is one of the first and most important jobs on any new machine.

How network servers handle clients

Network servers are daemons like cron, except that they listen on ports. Common examples are sshd (remote login), nginx, httpd or apache2 (web), postfix (mail), cupsd (printing), smbd (Windows-style file sharing) and nfsd (NFS). The classic design has one process listening on the port; for each new connection it calls fork(), and the child, often called a worker, serves that client and exits when the connection closes. Forking for every client costs time, so busy servers pre-create a pool of workers or use threads, and some, such as nginx, handle many connections in each worker with event-driven I/O. UDP servers have no connections, so they simply receive datagrams and answer them.

SSH and OpenSSH

SSH (Secure Shell) gives encrypted remote logins, remote command execution, file transfer and tunnelling of other connections. It replaced telnet, rlogin and rsh, which sent everything, passwords included, as plain text. OpenSSH is the implementation shipped by almost every distribution: the client is ssh and the server is sshd, listening on TCP port 22. Only protocol version 2 is supported. SSH uses public-key cryptography to authenticate the server and, optionally, the user, then encrypts the session with symmetric ciphers; the Networking topics "Principles of cryptography" and "End-point authentication" explain the ideas. On Debian and Ubuntu install the server with sudo apt install openssh-server; Fedora and RHEL usually include it.

Host keys and known_hosts

Each server has host key pairs in /etc/ssh, such as ssh_host_ed25519_key (private, readable only by root) and ssh_host_ed25519_key.pub (public). They are generated when the package is installed or when sshd first starts. On your first connection, ssh shows the server's key fingerprint and asks whether to trust it; if you accept, the key is stored in ~/.ssh/known_hosts. On later visits a different key triggers WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!, naming the offending line in known_hosts. That may be harmless, after a reinstall, or a sign of a man-in-the-middle, so confirm the new fingerprint with the administrator before running ssh-keygen -R host to remove the old entry. DSA host keys are obsolete: OpenSSH 10.0 (2025) removed DSA support entirely.

Configuring sshd

The server reads /etc/ssh/sshd_config (do not confuse it with ssh_config, the client's file). Lines are keyword-value pairs, and commented lines often show defaults. On current Debian, Ubuntu and Fedora the file begins with Include /etc/ssh/sshd_config.d/*.conf, and because sshd uses the first value it finds for each keyword, settings in those drop-in files win over later lines in the main file. Important keywords: Port, ListenAddress, PermitRootLogin, PasswordAuthentication, PubkeyAuthentication, AllowUsers and X11Forwarding. sudo sshd -t checks the configuration for errors before you apply it, and sshd -T prints the settings actually in force. The full list is in the sshd_config(5) manual page.

Running the server under systemd

The service unit is ssh.service on Debian and Ubuntu and sshd.service on Fedora and RHEL. sudo systemctl enable --now sshd starts it and enables it at boot on Fedora; after a configuration change, sudo systemctl reload ssh (or sshd) applies it without dropping existing sessions. Since Ubuntu 22.10 (2022), Ubuntu has started sshd on demand through ssh.socket, a systemd socket unit, to save memory. From Ubuntu 24.04 a generator copies Port and ListenAddress from sshd_config into the socket, so a change needs sudo systemctl daemon-reload followed by sudo systemctl restart ssh.socket (22.10 to 23.10 used a drop-in file for ssh.socket instead). Before systemd, the same on-demand idea was provided by inetd and later xinetd, superservers that listened on many ports and started a program per connection; both are now legacy.

Using the SSH client

ssh alice@server.example.com logs in; omit alice@ if the user names match, and use -p 2222 for a non-standard port. ssh server uptime runs one command and returns. Data can be piped through: tar czf - project | ssh server 'tar xzf -' copies a directory. Two file-copy tools come with OpenSSH: scp file alice@server:/tmp/ copies like cp, and sftp server opens an interactive session with get and put. Since OpenSSH 9.0 (2022), scp uses the SFTP protocol underneath by default. Per-host settings, such as a user name, port or key file for each server, belong in ~/.ssh/config. ssh -L 8080:localhost:80 server forwards local port 8080 through the encrypted connection to port 80 on the server. For large or repeated copies, rsync is better, as the topic "Quick copies and rsync" shows.

Worked example: setting up key-based login

Ana wants to log in from her laptop to server.example.com without a password. 1. On the laptop, create a key pair: ssh-keygen -t ed25519 -C "ana@laptop". Accept the path ~/.ssh/id_ed25519 and choose a passphrase; the output ends with a SHA256 fingerprint. 2. Copy the public key: ssh-copy-id ana@server.example.com asks for her password one last time and appends id_ed25519.pub to ~/.ssh/authorized_keys on the server. 3. Test: ssh ana@server.example.com now asks only for the key's passphrase. 4. Keep typing to a minimum: eval "$(ssh-agent)" then ssh-add loads the key once per session. 5. Never copy the private key, id_ed25519, to any other machine.

Worked example: permissions that SSH insists on

Key login fails, and the server's log (sudo journalctl -u ssh) mentions bad ownership or modes. By default sshd runs with StrictModes yes and ignores authorized_keys if the group or others can write to it or to ~/.ssh; 700 and 600 are the safest values. 1. Check: ls -ld ~/.ssh and ls -l ~/.ssh on the server. 2. Set the directory to 700 (owner rwx, nobody else): chmod 700 ~/.ssh shows drwx------. 3. Set authorized_keys to 600: chmod 600 ~/.ssh/authorized_keys shows -rw-------. 4. On the client, the private key must be readable only by its owner (600, or 400), or ssh refuses to use it with a warning that the key's permissions are too open; the .pub file can be 644 (-rw-r--r--). 5. Make sure the files are owned by the user, then try again.

Hardening sshd and fail2ban

Once key login works, keep a second session open, then set PasswordAuthentication no and PermitRootLogin no, run sudo sshd -t and reload. The open session lets you undo a mistake that would otherwise lock you out. Restrict who may log in with AllowUsers or AllowGroups. Any SSH server reachable from the internet sees constant automated password-guessing; with passwords off these attempts fail, but they still fill the logs. fail2ban watches the logs and, after a set number of failures from one address within a time window, adds a temporary firewall rule (through nftables or iptables) to block it. sudo fail2ban-client status sshd shows the banned addresses.

Common mistakes to avoid

First, editing ssh_config when you meant the server's sshd_config. Second, disabling passwords before testing key login, from your only session. Third, setting a value in sshd_config that a file in sshd_config.d has already set, then wondering why it is ignored. Fourth, leaving ~/.ssh or authorized_keys writable by others, so sshd silently rejects the key. Fifth, deleting a changed known_hosts entry without checking the new fingerprint. Sixth, copying a private key between machines instead of creating one per device.

Key takeaways

Servers listen on ports and fork, pre-fork or use threads and events to serve clients. OpenSSH provides ssh and sshd on TCP 22, using host keys checked against known_hosts. sshd_config and its sshd_config.d drop-ins configure the server; first value wins, sshd -t checks syntax and reload applies changes. Key login uses ssh-keygen -t ed25519, ssh-copy-id and authorized_keys, with modes 700 for ~/.ssh and 600 for authorized_keys and private keys. Then disable password and root logins, and let fail2ban block repeat offenders. scp, sftp and ~/.ssh/config complete the client's toolkit.

Flashcards

Flashcard

Worker process

Child that serves one connection for a server

Classic design: listen, accept, fork.

Test your understanding

Question 1 of 5

Key login fails, and the server log says authorized_keys has bad modes. Which settings fix it?

Question 2 of 5

You set PasswordAuthentication no at the end of sshd_config, reload, and passwords still work. What is the most likely reason?

Question 3 of 5

ssh warns REMOTE HOST IDENTIFICATION HAS CHANGED for a server you use daily. What should you do first?

Question 4 of 5

Which command copies your public key into the right file on a server?

Question 5 of 5

Why do busy servers often create a pool of worker processes at start-up instead of forking for each client?

← Previous topicNext topic →