Linux › Section K · Shell scripts › Topic K.6
Important shell script utilities
The shell itself is good at running programs and poor at editing text or doing sums, so scripts lean on a handful of small utilities. basename, awk, sed, xargs, shell arithmetic and exec cover most of what everyday scripts need.
Small tools that scripts rely on
Some utilities, such as basename, are rarely typed at the prompt but appear in script after script; others, such as awk and sed, are just as useful interactively. This topic gives the working core of each, not the whole language. The sample files are a users.txt in /etc/passwd format (seven colon-separated fields per line) for ada, bob and a service account svc, and a sizes.txt holding alpha 12, beta 30 and gamma 8. Outputs are real runs with GNU awk 5.0 and GNU sed 4.9.
basename and dirname
basename removes the directories from a path, and optionally a suffix: basename /usr/local/bin/backup prints backup, and basename photo.jpeg .jpeg prints photo. dirname does the opposite and keeps only the directory part: dirname /usr/local/bin/backup prints /usr/local/bin. Both work on text only; the path need not exist. Inside a script they usually appear in command substitution, as in b=$(basename "$f" .jpeg). For simple cases the shell can do the same without starting a program: with f=/home/ada/report.txt, ${f##*/} gives report.txt, ${f%.txt} gives /home/ada/report and ${f%/*} gives /home/ada.
Worked example: changing file extensions
Rename every .jpeg file in a directory to .jpg, in a directory holding beach.jpeg, old town.jpeg and notes.txt. 1. Loop over the glob: for f in *.jpeg; do. 2. Stop quietly if nothing matched: [ -e "$f" ] || exit 0. 3. Strip the suffix: b=$(basename "$f" .jpeg). 4. Report and rename: echo "renaming $f to $b.jpg", then mv -- "$f" "$b.jpg", then done. The -- tells mv that what follows are names, not options, in case a name starts with a dash. 5. Run it. It printed renaming beach.jpeg to beach.jpg and renaming old town.jpeg to old town.jpg, and ls showed beach.jpg, notes.txt and old town.jpg. A second run matches nothing and exits with status 0.
awk: printing fields
awk reads input line by line, splits each line into fields and runs a small program on it. Most scripts use it to pick out columns. Fields are $1, $2 and so on, $NF is the last field and NR is the line number. Whitespace separates fields by default; -F sets another separator. So awk '{print $2}' sizes.txt prints 12, 30 and 8, and awk -F: '{print $1, $7}' users.txt prints each user with their shell, such as ada /bin/bash. A condition before the braces selects lines: awk -F: '$3 >= 1000 {print $1}' users.txt prints ada and bob, the accounts with user IDs of 1000 or more, and awk -F: '$7 == "/bin/bash" {print $1}' prints ada. Single quotes protect the $ signs from the shell.
Worked example: totalling a column with awk
Add up the second column of sizes.txt. 1. For every line, add the field to a running total: {sum += $2}. awk variables start at 0, so no setup is needed. 2. Print once at the end, using the special END pattern, which runs after the last line: END {print "total", sum}. 3. Run awk '{sum += $2} END {print "total", sum}' sizes.txt. It printed total 50. 4. Check by hand: 12 + 30 + 8 = 50. 5. Beware values that contain spaces: a filename such as my notes.txt at the end of a line becomes two fields, so $NF gives only notes.txt.
sed: substitution
sed, the stream editor, applies editing commands to each line of its input and writes the result to standard output, leaving the file unchanged. Its workhorse is s/pattern/replacement/, where the pattern is a regular expression from "Intermediate commands: grep, less, diff, find, sort". sed 's/:/%/' users.txt replaces only the first colon on each line, giving ada%x:1000:..., while the g flag in s/:/%/g replaces every one. Any character can be the delimiter, which helps with paths: sed 's|/bin/sh$|/bin/bash|' changed bob's shell in the output. With -E for extended expressions, \1 and \2 recall bracketed groups: sed -E 's/([a-z]+) ([0-9]+)/\2=\1/' sizes.txt printed 12=alpha, 30=beta and 8=gamma.
sed: addresses, deleting and printing
A command can be limited to certain lines by an address in front of it, a line number, a range or a /regex/. sed '2d' deletes line 2, sed '3,6d' deletes lines 3 to 6, and sed '/nologin/d' users.txt deleted the svc line, leaving ada and bob. With -n, sed prints only what you ask for with p: sed -n '2p' printed bob's line, and sed -n '/bash$/p' printed ada's. To change a file in place, GNU sed has -i; give it a suffix, as in sed -i.bak 's/beta/BETA/' s2.txt, and it keeps the original as s2.txt.bak. Test the expression without -i first, because a wrong in-place edit overwrites the file with no undo.
xargs and find -exec
xargs, met in "Shell input and output: redirection and pipes", runs a command with words from standard input as its arguments. printf 'a\nb\nc\n' | xargs echo printed a b c, and with -n 1 it ran echo once per item. A common pairing is find with xargs, but plain newline-separated names break on spaces: find pics -name '*.gif' | xargs ls split my cat.gif into my and cat.gif. The safe form uses null separators on both sides: find pics -name '*.gif' -print0 | xargs -0 ls listed all three files correctly. find can also run the command itself: -exec cmd {} \; runs once per file, and -exec cmd {} + passes many names to one run, like xargs.
Arithmetic: $(( )) and expr
The POSIX shell does integer arithmetic inside $(( )): with n=5, echo $((n * 3)) prints 15, and n=$((n + 1)) makes n 6. Variables need no $ inside, brackets group as usual and % gives a remainder. Division discards the fraction: $((7 / 2)) is 3, and $((10 / 3 * 3)) is 9, not 10. $((1.5 + 1)) fails with a syntax error, so use awk or Python for fractions. bash adds ** for powers. The older expr command does the same job slowly and needs spaces and escaping: expr 7 \* 6 prints 42, but expr 1+2 just prints 1+2.
exec
The exec builtin replaces the current shell with another program, using the exec() system call from "The kernel: processes, memory, drivers, and system calls". No new process is created and there is no return. A test script printed before exec, pid 20245, then exec started a new bash that printed now running in pid 20245 (the number varies by run), the same process; the script's last line never ran. Wrapper scripts use exec as their last line, such as exec /opt/app/bin/server "$@", so the real program takes over the process, receives signals directly and leaves no idle shell behind.
Common mistakes to avoid
First, writing awk programs in double quotes, so the shell expands $1 before awk sees it. Second, forgetting -F: for colon-separated files. Third, expecting sed to change the file without -i, or using -i untested and without a backup suffix. Fourth, forgetting the g flag when every match should change. Fifth, piping find into xargs without -print0 and -0, which breaks on spaces. Sixth, expecting $(( )) to handle decimals, or forgetting that integer division truncates. Seventh, putting commands after exec and expecting them to run.
Key takeaways
basename strips directories and suffixes and dirname keeps the directory. awk prints fields: $1, $NF, NR, -F to set the separator, a condition to select lines and END for totals. sed edits a stream: s/old/new/ and the g flag, addresses with d and -n with p, and -i.bak for an in-place edit with a backup. find ... -print0 | xargs -0 cmd, or -exec cmd {} +, handles any filename. $(( )) does integer arithmetic, with expr as the slow old alternative. exec replaces the shell with another program.
Flashcards
Flashcard
Removes the directories and the given suffix
Test your understanding
Question 1 of 5
users.txt has colon-separated fields with the login name first and the user ID third. Which command prints the names of users whose ID is 1000 or more?
Question 2 of 5
A line reads a:b:c. What does sed 's/:/-/' print for it?
Question 3 of 5
What does echo $((10 / 3 * 3)) print in bash?
Question 4 of 5
A directory holds three .gif files. How many times does find . -name '*.gif' -exec ls {} + run ls?
Question 5 of 5
A script ends with exec /opt/app/bin/server "$@" followed by echo done. What happens?
