Linux › Section B · Basic commands and the directory hierarchy › Topic B.4

Intermediate commands: grep, less, diff, find, sort

Once you can move around and manage files, the next step is finding things: lines in a log, a file lost somewhere in a tree, the change between two versions. This topic covers the standard tools for searching, viewing, comparing and counting text, with a short primer on regular expressions.

Tools that work on text

Most Linux configuration files and logs are plain text, so a few text tools go a long way. Each does one job, and they can be chained: the | symbol, a pipe, sends one command's output into the next, as the topic "Shell input and output: redirection and pipes" explains in full. The examples here use a five-line log, app.log, whose second field is a level such as INFO or ERROR, and were run with GNU tools in bash 5.2.

grep: finding matching lines

grep prints every line of its input that matches a pattern: grep ERROR app.log prints the lines containing ERROR. Matching is case-sensitive unless you add -i. The -v option inverts the search, printing lines that do not match; -n adds line numbers and -c prints only a count. Given several files, grep puts the file name before each match, as in web.log:10:00 ERROR 404, which makes it ideal for searching many files at once, such as grep -i error *.log. With no file arguments it reads standard input, so it can filter another command's output.

A short regular-expression primer

grep's patterns are regular expressions, a pattern language different from shell globs. In a regular expression, . matches any one character and * means zero or more of the item before it, so .* matches any run of characters, including none. ^ anchors a match to the start of a line and $ to the end. Square brackets match one character from a set, as in [0-9] for a digit. So ^09:0[13] matches lines beginning 09:01 or 09:03. Put patterns in single quotes, as in grep 'E.*R' app.log, so the shell does not treat * as a glob and expand it before grep sees it.

Basic and extended regular expressions

grep understands two dialects. By default it uses basic regular expressions (BRE), where +, ?, | and parentheses are ordinary characters. With grep -E it uses extended regular expressions (ERE), where + means one or more of the previous item, ? means zero or one, | separates alternatives and parentheses group. So grep -E 'colou?r' matches color and colour, while plain grep 'colou?r' looks for a literal question mark and matched neither. GNU grep also accepts \? and \+ in BRE as an extension. The old command egrep means grep -E; since GNU grep 3.8 (2022) it prints a warning, so write grep -E.

Worked example: searching a log

app.log holds: 09:00 INFO start, 09:01 ERROR disk full, 09:02 info retry, 09:03 Error timeout and 09:04 INFO done. 1. grep ERROR app.log printed one line, 09:01 ERROR disk full, because matching is case-sensitive. 2. grep -i error app.log printed two lines, the 09:01 and 09:03 entries. 3. grep -v -i info app.log printed the same two lines: every line without info in any case. 4. grep -c INFO app.log printed 2. 5. grep -E 'full|done' app.log printed the 09:01 and 09:04 lines. Check: the count in step 4 excluded the lowercase info line, consistent with step 1's case sensitivity.

less, head and tail

less shows a file one screen at a time: less /etc/services, or any long output piped into it. Press the space bar to go forward, b to go back and q to quit. Type /word and Enter to search forward, ?word to search backward, and n to jump to the next match. It improves on the older more. To see only part of a file, head -n 2 app.log prints the first two lines and tail -n 1 app.log the last one; both show 10 lines by default. tail -n +4 starts at line 4, and tail -f keeps printing new lines as a log grows.

diff and file

diff old.txt new.txt shows how to turn the first file into the second. With old.txt holding apples, bread, milk and new.txt holding apples, butter, milk, eggs, diff printed 2c2 (change line 2), < bread, ---, > butter, then 3a4 (after line 3, add line 4) and > eggs. diff -u gives the unified format that programmers share and patch tools read: unchanged lines start with a space, removed lines with - and added lines with +. diff exits with status 0 when the files are identical. When you meet a file of unknown kind, file name inspects its contents and reports, for example, app.log: ASCII text.

find, locate and pwd -P

find searches a directory tree as it is right now: find docs -name '*.txt' -print lists every file under docs whose name ends in .txt. -print is the default action, so it may be left out, and the order of results follows the directory layout rather than any sorting. Quote the pattern so the shell does not expand it first. locate name searches a prebuilt index instead, which is much faster, but it misses files created since the index was last updated. When symbolic links are involved, pwd may print the path you followed; pwd -P prints the physical path.

sort, cut, uniq, wc and tr

sort puts lines in order; -n sorts numerically and -r reverses. Plain sort compares text, so 10 pears came before 9 figs, while sort -n gave 9, 10, 100. cut extracts parts of each line: cut -d ' ' -f 2 takes field 2, using a space as the delimiter. uniq collapses adjacent identical lines, and -c prefixes a count; because only neighbours are merged, it normally follows sort. wc counts lines, words and bytes: wc app.log printed 5 16 92 app.log, and wc -l gives lines alone. tr translates characters: tr a-z A-Z converts lower case to upper case.

Worked example: counting log levels

How many lines of app.log has each level, ignoring case? 1. cut -d ' ' -f 2 app.log printed INFO, ERROR, info, Error, INFO, one per line. 2. Piping that into tr a-z A-Z made them INFO, ERROR, INFO, ERROR, INFO. 3. sort grouped identical lines together. 4. uniq -c counted each group. The full command was cut -d ' ' -f 2 app.log | tr a-z A-Z | sort | uniq -c, and it printed 2 ERROR and 3 INFO. Check: 2 + 3 = 5, matching wc -l, and without tr the counts split into four groups because case differed.

Common mistakes to avoid

First, forgetting that grep is case-sensitive without -i. Second, leaving a pattern unquoted, so the shell expands * or ? before grep or find sees it. Third, using + or | without -E, where basic syntax treats them as ordinary characters. Fourth, mixing up globs and regular expressions: in a regular expression * repeats the previous item, it does not match anything alone. Fifth, running uniq on unsorted input. Sixth, using plain sort on numbers. Seventh, trusting locate for brand-new files.

Key takeaways

grep prints matching lines: -i ignores case, -v inverts, -n numbers, -c counts, and several files are labelled by name. Regular expressions use . for any character, * for repetition, ^ and $ as anchors and [ ] for sets; -E enables +, ?, | and grouping, and replaces egrep. less pages through text; head and tail show the start or end. diff compares files, -u in unified form. find searches the tree live; locate uses an index. sort (-n, -r), cut (-d, -f), uniq (-c, after sort), wc and tr reshape and count text.

Flashcards

Flashcard

grep pattern file

Prints lines of the file matching the pattern

Several files: names are shown.

Test your understanding

Question 1 of 5

Which command prints the lines of app.log that contain neither error nor ERROR nor Error?

Question 2 of 5

A file contains the lines color and colour. What does grep 'colou?r' file print, using grep's default syntax?

Question 3 of 5

A file lists the lines 10, 9 and 100. What order does plain sort give?

Question 4 of 5

Why does a command such as cut -f 1 data | uniq -c usually need sort before uniq?

Question 5 of 5

You created report.txt a minute ago. Which command is sure to find it under your home directory?

← Previous topicNext topic →