Linux › Section L · Network file transfer and sharing › Topic L.1
Quick copies and rsync
Moving files between machines ranges from grabbing one file once to keeping two directory trees identical every night. A throwaway web server, scp and sftp cover the quick jobs, and rsync, the standard synchroniser on Linux, handles the rest efficiently and safely.
Picking a way to move files
For a single file fetched once on your own network, a temporary web server is enough. For a few files to a machine you can log in to, scp or sftp over SSH does it. For a large tree, a regular copy or an exact mirror, use rsync, which sends only what has changed. Attaching a remote directory so it behaves like a local one is file sharing, covered in "File sharing with Samba" and "SSHFS, NFS, and cloud storage". The rsync examples here follow the rsync manual; any output shown is illustrative.
A quick copy with Python's web server
Python 3 includes a tiny web server. Run python3 -m http.server in a directory and a browser on another machine can download its files from http://10.1.2.4:8000/, if your address is 10.1.2.4. The default port is 8000; a number after the command chooses another, --directory serves a different directory, and --bind 127.0.0.1 limits it to the local machine, because by default it listens on all interfaces. Ctrl+C stops it. Old guides show Python 2's obsolete python -m SimpleHTTPServer. Warning: anyone who can reach the port can read the files without a password, and Python's documentation says it is not for production. Use it only briefly on a network you trust.
scp and sftp
Both ride on SSH, covered in "Network servers and SSH", so they need only an SSH server at the other end. scp report.pdf ada@server:docs/ copies a file into docs under ada's home directory, scp ada@server:logs/app.log . copies one back, and scp -r photos ada@server: copies a whole directory. sftp ada@server opens an interactive session with commands such as ls, cd, get and put. Since OpenSSH 9.0, released in April 2022, scp itself uses the SFTP protocol by default, with -O to force the legacy protocol for old servers. scp suits one-off copies, but it resends everything each time and never removes destination files deleted at the source, so copies drift.
rsync basics
rsync must be installed on both machines and normally connects over ssh. Its syntax echoes scp: rsync file1 file2 ada@server: copies two files to ada's home directory there; --rsync-path names the remote rsync if it is not on the PATH. Given a directory without options, rsync skips it. The usual choice is -a, archive mode, which the manual defines as equal to -rlptgoD: recursive, copying symbolic links as links and preserving permissions, modification times, group, owner (when run as root) and device files. -a does not include hard links (-H), ACLs (-A) or extended attributes (-X). Put the remote path first to pull files to the local machine, rsync -a server:src dest, or name no host at all to copy between local directories.
Dry runs and the quick check
When unsure what a command will do, add -n (--dry-run) and -v (verbose): rsync -nva dir server:dest lists what would be transferred without changing anything. By default rsync decides that a file is unchanged when its size and modification time match at both ends, the quick check. A second run of the same command therefore lists no files, which proves the copy is complete. -c (--checksum) compares file contents instead, slower but stricter. --stats prints a summary, and -P shows progress and keeps partly sent files so an interrupted copy can resume.
Worked example: the trailing slash
A local directory dir holds files a and b, and the remote directory dest is empty. 1. Run rsync -a dir server:dest. rsync copies the directory itself, giving dest/dir/a and dest/dir/b. 2. Empty dest again and run rsync -a dir/ server:dest. The trailing slash means the contents of dir, so the result is dest/a and dest/b, with no dir level; the manual says the slash avoids creating an additional directory level. 3. Compare with cp: the slash form behaves like cp dir/* dest, the plain form like cp -r dir dest. 4. Before a real run, rsync -nva with either form shows which paths would be created.
Exact mirrors with --delete
By default rsync adds and updates files but never removes them, so if dest/dir already holds a file c that is no longer in dir, c survives. --delete removes such extra files, making the destination an exact copy: rsync -a --delete dir/ server:dest. The manual warns that this option can be dangerous if used incorrectly and recommends a --dry-run first to see what would be deleted. With a misplaced slash or a wrong destination, everything there that is not in the source is removed, and tab completion silently adds trailing slashes, so reread the command before pressing Enter.
Excluding files and other safeguards
--exclude=PATTERN skips matching names, and can be repeated: rsync -a --exclude=.git src server: leaves out every .git anywhere in the tree, because a pattern without a slash is matched against the last part of each path. A leading slash anchors it to the top of the transfer, not the system root, so --exclude=/src/.git skips only that one. A trailing slash, as in --exclude=cache/, matches directories only. --exclude-from=FILE reads one pattern per line. To protect the destination, --ignore-existing skips files already there, -u skips files newer there, and -b renames replaced files with a ~ suffix. -z compresses data, which helps on slow links but can slow a fast LAN. --bwlimit caps the rate; per the manual a plain number means units of 1024 bytes per second, so --bwlimit=5000 is about 5 MB/s.
Worked example: a nightly backup with a dry run
Mirror ~/projects to /srv/backup/projects on the host backup. 1. Write the command with the slash on the source: rsync -a --delete --exclude=node_modules/ ~/projects/ backup:/srv/backup/projects/. 2. Run it first with -nv added. The listing shows the files to send and lines beginning deleting for anything removed at the destination; read them all. 3. Run it for real, then again with -v; the second run should list no files, as all pass the quick check. 4. Automate it with a schedule from "Recurring tasks with cron and timer units". With --delete, a file deleted by mistake in ~/projects vanishes from the mirror at the next run: a mirror alone is not a backup.
Backup strategy: the 3-2-1 rule
A mirror protects against a dead disk, not against deletion, corruption or ransomware, which it faithfully copies. A widely used rule of thumb is 3-2-1: keep at least three copies of important data, on at least two different kinds of storage, with at least one copy off site, away from the building. Keep versions, not just the latest state: rsync's --link-dest option can build dated snapshots that hard-link unchanged files to the previous snapshot, so each day costs only the space of what changed. Above all, test restores regularly: a backup never restored from is only a hope.
Common mistakes to avoid
First, serving files with python3 -m http.server on an untrusted network, or leaving it running. Second, expecting scp to remove files deleted at the source. Third, forgetting -a, so rsync skips directories. Fourth, confusing dir and dir/ as the source, especially with --delete. Fifth, running --delete without a dry run. Sixth, assuming --exclude=/x refers to the system's root. Seventh, treating a mirror as a backup.
Key takeaways
python3 -m http.server shares a directory on port 8000 for a quick copy on a trusted network. scp and sftp copy over SSH; since OpenSSH 9.0 scp uses the SFTP protocol. rsync -a (equal to -rlptgoD) copies trees, sending only files that fail the size-and-time quick check. A trailing slash on the source copies its contents, not the directory. --delete makes an exact mirror, so test it with -nv first. --exclude skips patterns, -z compresses, --bwlimit caps the rate in KiB/s. Follow the 3-2-1 rule and test restores.
Flashcards
Flashcard
Serves the current directory over HTTP on port 8000
Test your understanding
Question 1 of 5
The local directory dir contains a and b. After rsync -a dir/ server:dest, where are the files on the server?
Question 2 of 5
Which command shows what an rsync mirror with deletion would do, without changing anything?
Question 3 of 5
By default, how does rsync decide that a file at the destination is already up to date?
Question 4 of 5
A nightly job runs rsync -a --delete ~/work/ backup:/srv/work/. You delete a file by mistake in the afternoon. What happens that night?
Question 5 of 5
Since OpenSSH 9.0, released in 2022, what protocol does scp use by default?
