Linux › Section K · Shell scripts › Topic K.8
When (not) to use shell scripts
Knowing a tool includes knowing its limits. Shell scripts are superb glue for running programs and handling files, but some jobs make them slow, fragile and hard to read, and recognising those jobs early saves hours of debugging.
What the shell does best
This section has shown the shell's strengths: running programs, connecting them with pipes, testing their exit codes, looping over files and making small edits with sed and awk. A script that backs up a directory, rotates logs, wraps a command with standard options or checks that a service answers plays to all of them. Such scripts are short, need nothing installed beyond the base system, and read almost like the commands an administrator would type. The trouble starts when a script stops mostly running other programs and starts doing the work itself: parsing text, calculating, keeping track of structured data.
Warning signs
Several symptoms suggest a script has outgrown the shell. It is getting long: Google's public Shell Style Guide advises rewriting any script over 100 lines, or with non-straightforward control flow, in a more structured language. It picks strings apart character by character. It needs decimals, which $(( )) cannot do. It tracks records with several fields each, a job for real data structures. It asks the user many questions with read. Its error handling has grown into a maze of checks on $?. Or it must be fast. None of these is forbidden, but each makes the code harder for the next person to maintain, and that person is often its author six months later.
Worked example: a CSV file defeats awk
A file scores.csv has a header line name,city,score and two records: "Lovelace, Ada",London,91.5 and Hopper,New York,88. You want the scores. 1. The obvious tool is awk -F, 'NR > 1 {print $3}' scores.csv. 2. It printed London and 88. The second line is right, but the first is wrong. 3. The reason: CSV allows a comma inside a quoted field, and awk -F, splits at every comma, so "Lovelace became field 1, Ada" field 2, and the city slid into field 3. 4. Handling quotes properly in sh or awk means writing a small parser, exactly the fiddly string work the warning signs describe. 5. The lesson generalises: a format with quoting or nesting rules, such as CSV, JSON or XML, deserves a real parser.
Worked example: the same job in Python
1. Python 3, available on almost every Linux distribution (3.11 was used here), has a csv module that understands quoting. A six-line script opens scores.csv, reads it with csv.DictReader so each row becomes a mapping from column name to value, and converts the scores with float. 2. Printing rows[0]["name"] and rows[1]["city"] gave Lovelace, Ada | New York, both correct. 3. The average, sum(scores) / len(scores), printed 89.75. 4. Check: (91.5 + 88) / 2 = 179.5 / 2 = 89.75. The shell could not have done step 3 alone: $(( )) has no decimals, and awk 'BEGIN {print (91.5 + 88) / 2}' would be needed just for the arithmetic. 5. The Python version is also easier to extend, say to skip blank scores or write a report.
The cost of starting processes
Every external command a script runs, such as basename, sed or a $( ) around one of them, starts a new process, and process start-up is slow compared with work done inside the shell. On the test machine for this course, a 1000-pass loop running b=$(basename /a/b/file.txt) took tens of seconds (the exact time varied between runs), while the same loop using the shell's own ${f##*/} took under a tenth of a second. That machine ran bash under Windows, where starting processes is unusually slow, so a Linux system is much faster and narrows the gap, but the direction holds everywhere. A loop that calls several programs per line of a large file will crawl. awk or Python handle the whole file in one process.
Choosing the alternative
Match the tool to the job. For tables of text, awk often suffices and is on every system. For anything with real logic, data structures, decimals, networking or careful error handling, Python is the usual choice today: its standard library includes subprocess to run commands, pathlib for files, csv and json for data formats, and argparse for options. Perl, once the standard step up from shell, still runs many older system scripts. When raw speed matters, a compiled language such as C, Go or Rust may be justified. A common split is a thin shell wrapper that sets the environment and then runs exec on a Python program, as "Important shell script utilities" described.
Configuration management with Ansible
A different limit appears when a script must set up many machines identically: install packages, copy configuration files, start services. A shell script runs steps blindly, so running it twice can append a line twice or fail half-way and leave a machine in an unknown state. Configuration management tools instead describe the state you want. Ansible, a widely used open-source example, is agentless: from a control node it connects to the managed nodes over SSH with existing credentials, so no agent runs on them, though most modules need Python there. An inventory lists the managed nodes; a playbook, written in YAML, holds plays that map hosts to tasks; each task calls a module, the code Ansible copies to the node and runs there.
Idempotence: why Ansible is not just a script
Ansible's tasks are written as desired states, not actions. A task using the module ansible.builtin.package with name: nginx and state: present says nginx should be installed, and one using ansible.builtin.service with state: started and enabled: true says it should be running and start at boot. If the machine already matches, the task changes nothing. This property is called idempotence: the result is the same however many times a playbook runs. You would run such a playbook as ansible-playbook -i inventory.ini site.yml. Puppet, Chef and Salt are other tools in the same family. For one machine and a few commands a shell script remains simpler; for a fleet, a declarative tool is safer.
Common mistakes to avoid
First, growing a quick script for years until it is hundreds of lines nobody dares touch. Second, parsing CSV, JSON or other quoted formats with cut, awk -F or sed. Third, doing decimal arithmetic in the shell, or pulling in expr and bc for every sum. Fourth, calling external programs inside a loop over a large file. Fifth, the opposite error: writing a large Python program where a five-line pipeline would do. Sixth, managing many servers with scripts that are not safe to run twice.
Key takeaways
Shell scripts excel at running programs, piping them together and handling files, and they should stay short. Move to another language when a script grows long or complex, parses structured text, needs decimals or data structures, or must be fast. awk handles simple tables; Python, with modules such as csv, json, pathlib and subprocess, handles almost everything else; compiled languages serve when speed is critical. Each external command costs a process. For configuring many machines, Ansible playbooks describe a desired state over SSH and are idempotent, so they are safe to re-run.
Flashcards
Flashcard
Running programs, pipes, exit codes and file handling
Test your understanding
Question 1 of 5
Which job is the best fit for a short shell script?
Question 2 of 5
A CSV line reads "Lovelace, Ada",London,91.5. What does awk -F, '{print $3}' print for it?
Question 3 of 5
Why does a loop over 10,000 lines that runs sed and basename on every line tend to be slow?
Question 4 of 5
What does it mean that an Ansible playbook is idempotent?
Question 5 of 5
How does Ansible usually reach the Linux machines it manages?
