Linux › Section B · Basic commands and the directory hierarchy › Topic B.8

Shell input and output: redirection and pipes

Every command reads from one place and writes to another, and the shell lets you choose both. Redirection connects commands to files, and pipes connect commands to each other, so small tools combine into powerful one-line solutions.

Three standard streams

Every process starts with three open streams, each identified by a small number called a file descriptor. Standard input (stdin, 0) is where the program reads its input. Standard output (stdout, 1) is where it writes its normal results. Standard error (stderr, 2) is a separate channel for error messages and diagnostics. In an interactive shell all three are connected to your terminal, so you type into stdin and see both stdout and stderr on screen. Because errors travel on their own stream, you can save or process a command's results while its error messages still reach you.

Sending output to a file

The > operator sends stdout to a file instead of the terminal: ls /etc > listing.txt. The shell creates the file if it does not exist and, if it does, empties it first, so the old contents are lost. To add to the end of a file instead, use >>: date >> run.log appends one line each time it runs. Bash can protect you from accidental clobbering: after set -C (the noclobber option), > refuses to overwrite an existing file and reports cannot overwrite existing file, while >| overwrites deliberately. set +C turns the protection off.

Redirecting standard error

Run ls /nosuchdir > out.txt and the message ls: cannot access '/nosuchdir': No such file or directory still appears, and out.txt is empty, because > only redirected stream 1. Put the stream number in front of the operator to redirect another stream: ls /nosuchdir 2> err.txt sends the error into err.txt. You can separate the two completely, as in command > out.txt 2> err.txt. To throw output away, send it to /dev/null, a special file that discards everything written to it: command 2>/dev/null hides the error messages but keeps the normal output on screen.

Combining the two streams

To capture everything in one file, write command > all.txt 2>&1. The 2>&1 means make stream 2 point to wherever stream 1 points now. The shell processes redirections from left to right, so order matters: first stdout goes to all.txt, then stderr is pointed at the same place. Bash also accepts the shorthand command &> all.txt, which does the same job but is not part of the POSIX standard, so portable scripts use the longer form. To append both streams to a log, use command >> all.txt 2>&1.

Worked example: why order matters

Compare two commands run in a directory that contains files, using ls . /nosuchdir, which lists one real and one missing directory. 1. ls . /nosuchdir > both.txt 2>&1: stdout is pointed at both.txt, then stderr is copied from stdout, so both the listing and the error land in the file and the terminal shows nothing. 2. ls . /nosuchdir 2>&1 > only.txt: stderr is first copied from stdout, which is still the terminal, and only then is stdout moved to the file. 3. Result of the second command: the error appears on screen and only.txt holds just the listing. 4. Check with cat on each file.

Reading input from a file

The < operator connects a file to a command's stdin: sort < names.txt. Most commands also accept file names as arguments, so sort names.txt gives the same output and is more common. Input redirection matters for programs that only read stdin, and for commands like wc -l < names.txt, which prints just the count without the file name. A frequent trap is sort names.txt > names.txt: the shell empties names.txt before sort even starts, so sort reads an empty file and you lose the data. Write to a new file instead, or use an option such as sort -o names.txt names.txt.

Pipes

A pipe, written |, connects the stdout of one command to the stdin of the next: ls /usr/bin | wc -l counts the programs in /usr/bin without any temporary file. You can chain as many stages as you like, as in grep ERROR app.log | sort | uniq -c, and all the commands in a pipeline run at the same time, with data flowing between them as it is produced. Only stdout goes through a pipe. Error messages still go to the terminal unless you add 2>&1 before the pipe, as in make 2>&1 | less. Tools that read stdin and write stdout, called filters, are designed to sit in the middle of pipelines.

tee: save and see

tee copies its stdin to stdout and to one or more files at the same time. grep ERROR app.log | tee errors.txt | wc -l saves the matching lines in errors.txt and still passes them on to wc, which counts them. Use tee -a to append instead of overwriting. tee also solves a common permissions problem: in sudo echo text > /etc/file, the redirection is done by your own unprivileged shell, so it fails. Writing echo text | sudo tee /etc/file works, because tee runs as root and opens the file itself. The topic Running commands as the superuser covers sudo.

xargs: input into arguments

Some commands, such as rm and touch, ignore stdin and only act on arguments. xargs bridges the gap: it reads words from stdin and adds them to the end of a command line, then runs it. printf 'a.tmp b.tmp' | xargs touch creates both files. Option -n 1 runs the command once per word, and -I{} puts each input line where {} appears. Because xargs splits on spaces, a file called my file.tmp becomes two words, my and file.tmp. The safe pattern for file names is find . -name '*.tmp' -print0 | xargs -0 rm, where both sides separate names with a null character instead of spaces.

Worked example: a log summary

A file app.log holds lines such as ok start, ERROR disk, ok run and ERROR net, and you want the error lines saved and counted. 1. Select the lines: grep ERROR app.log. 2. Save a copy while passing them on: add | tee errors.txt. 3. Count them: add | wc -l. 4. The full pipeline is grep ERROR app.log | tee errors.txt | wc -l, which prints 2 in this example. 5. Check: cat errors.txt shows ERROR disk and ERROR net. 6. To keep a running record, change the last stage to wc -l >> counts.log, so each run appends a new count instead of erasing the old ones.

Common mistakes to avoid

First, using > when you meant >>, which wipes the file; set -C guards against this. Second, redirecting into the file you are reading, as in sort f > f, which empties it first. Third, writing 2>&1 before > file and wondering why errors still appear. Fourth, expecting a pipe to carry error messages: only stdout flows through | unless you add 2>&1. Fifth, writing sudo command > /etc/file, where the redirection runs without root; pipe into sudo tee instead. Sixth, feeding names with spaces to xargs without -print0 and -0.

Key takeaways

Processes have stdin (0), stdout (1) and stderr (2), all on the terminal by default. > writes stdout to a file, emptying it first; >> appends; set -C blocks accidental overwrites. 2> redirects errors, /dev/null discards output, and > file 2>&1 captures both streams, with order mattering. < feeds a file to stdin. A pipe | joins one command's stdout to the next's stdin, and the stages run together. tee saves a copy of a stream while passing it on, and xargs turns input into arguments, with -print0 and -0 for safe file names.

Flashcards

Flashcard

stdin, stdout, stderr

File descriptors 0, 1 and 2

All on the terminal by default.

Test your understanding

Question 1 of 5

Which command saves both the normal output and the error messages of ./build into build.log?

Question 2 of 5

A colleague runs sort users.txt > users.txt. What happens?

Question 3 of 5

Running sudo echo 'nameserver 1.1.1.1' > /etc/resolv.conf gives Permission denied. Why?

Question 4 of 5

What does grep ERROR app.log | tee errors.txt | wc -l do?

Question 5 of 5

Why is find . -name '*.tmp' -print0 | xargs -0 rm safer than find . -name '*.tmp' | xargs rm?

← Previous topicNext topic →