Linux › Section O · Development tools › Topic O.3
Lex, Yacc, scripting languages, and Java
C is only part of the toolkit on a Linux system. Parser generators, scripting languages such as Python and Perl, the Java runtime and the Git version-control system all turn up in everyday administration, and each has a few conventions worth knowing.
Beyond the C compiler
The previous two topics, "The C compiler and linking" and "make", covered how compiled programs are built. Much of a working system, however, is written in other ways. Some programs contain parsers generated by tools, many administration utilities are scripts, some applications run on the Java virtual machine, and almost every project's source now lives in a Git repository. You need not master them, but you should recognise them and know what runs them.
Lex and Yacc
Programs that read a configuration language or a command language need two pieces. A lexer splits raw text into tokens such as numbers, names and symbols; a parser checks that the tokens follow a grammar and acts on them. Lex and Yacc generate these pieces in C from a description. On Linux the free reimplementations are used: flex for Lex, which writes lex.yy.c, and bison for Yacc, which writes a .tab.c file (bison -y imitates Yacc and writes y.tab.c). When building such software you may need -lfl (the flex library, which supplies helper functions such as yywrap and a default main) or -ly (the Yacc library) on the link line.
How a script runs
A script is a text file with execute permission whose first line, the shebang, starts with #! followed by an interpreter path, such as #!/bin/sh or #!/usr/bin/python3. When you run it, the kernel starts that interpreter and passes it the script's path as an argument, so the interpreter reads the file itself. #!/usr/bin/env python3 instead uses whichever python3 comes first in PATH, useful when the interpreter's location varies. Linux passes everything after the interpreter path as one single argument, so rely on at most one option on the shebang line.
Worked example: testing shebang lines
1. Create a file t2 whose first line is #!/usr/bin/tail -n2, followed by three lines: line one, line two, line three. 2. chmod +x t2 and run ./t2. In a bash 5.2 test, the output was line two and line three. tail received -n2 and the script's own path, so it printed the file's last two lines, the shebang never being among them. 3. Now give a script the first line #!/usr/local/bin/nothere and run it. It fails. On Linux, bash reports bad interpreter: No such file or directory with exit status 126; Git Bash on Windows said cannot execute: required file not found, with status 127. Either way, check the path after #!.
Python
Python is now the most common general scripting language on Linux, used for system tools, automation, data analysis and web services. Python 2 reached end of life in January 2020, so the interpreter is python3; some distributions provide no plain python command at all unless you install one (on Debian and Ubuntu, the python-is-python3 package). For administration, the standard library covers most needs: pathlib and shutil for files, subprocess to run commands, os for the environment. For third-party packages, create a virtual environment with python3 -m venv. Debian 12 and Ubuntu 23.04 and later refuse a system-wide pip install by default, to protect packages the distribution manages.
Worked example: a small admin script
Goal: report how full the root filesystem is. 1. Write df.py with the shebang #!/usr/bin/env python3. 2. Import the standard module: import shutil. 3. Unpack three values: total, used, free = shutil.disk_usage("/"). 4. Print a summary: print(f"{used / total:.0%} used, {free // 2**30} GiB free"). The :.0% format turns a fraction into a whole-number percentage, and dividing by 2**30 gives binary gibibytes. 5. Run python3 df.py. Tested with Python 3.11, it printed 90% used, 22 GiB free. 6. Compare with the shell tool df -h /, which reports the same filesystem in human-readable units.
Perl and other languages
Perl, one of the oldest Unix scripting languages, remains strong at text processing and file conversion, and many older system tools are written in it. You will meet others: Ruby, PHP for web pages, JavaScript through Node.js (the node command), Tcl with its Tk toolkit and Expect automation tool, Lua as an embedded configuration language, R for statistics, and m4, a macro language found mainly inside GNU Autoconf. Newer compiled languages, especially Go and Rust, are now common for system software; the Linux kernel has accepted Rust code since version 6.1 in 2022.
Java
Java source is compiled to bytecode, stored in .class files, which the Java virtual machine (JVM) runs on any platform; this kind of virtual machine is a language runtime, not the hypervisor kind in "Virtual machines". To build, you need a JDK, usually an OpenJDK package. javac Hello.java writes Hello.class, and java Hello runs it, naming the class without its .class suffix. Applications often ship as .jar archives of classes, run with java -jar app.jar. Since Java 11, java Hello.java can also run a single source file directly. JAVA_HOME names the installation, and CLASSPATH, a colon-separated list like PATH, tells the JVM where to find classes.
Git basics
Git is a version-control system: it records snapshots of a set of files so you can see every change, return to any earlier state and work on alternatives safely. A repository is a project directory plus its hidden .git directory holding the history. You edit files in the working tree, choose which changes go into the next snapshot with git add (the staging area), and record them with git commit. git status and git diff show what has changed; git log lists commits. git clone copies a repository, git pull and git push exchange commits with a remote one, and branches (git switch -c name) hold separate lines of work.
Worked example: tracking a config file
Put a shared profile under version control. 1. git init -b main creates the repository. 2. git config user.name and user.email set the author recorded with each commit. 3. Write profile.sh containing a PATH line; git status --short shows ?? profile.sh, an untracked file. 4. git add profile.sh then git commit -m "Add base profile" records the first commit. 5. Append umask 022; git status --short now shows M profile.sh, and git diff shows the added line marked +umask 022. 6. git commit -am "Set umask" stages and commits tracked changes. 7. git log --oneline lists both commits, newest first, each with a short hash. All of this was run with Git 2.46.
Common mistakes to avoid
Writing a shebang with a path that does not exist on the target system, or with Windows line endings, which make the kernel look for an interpreter named python3 followed by a carriage return. Putting two options on a shebang line. Typing python and getting nothing, or the wrong version, instead of python3. Using pip as root to install into the system Python rather than a virtual environment. Running java Hello.class instead of java Hello. Committing passwords or keys into a Git repository, where they stay in the history even after the file is deleted. Expecting a link to succeed without -lfl or -ly.
Key takeaways
flex and bison generate lexers and parsers in C; link with -lfl or -ly when needed. A script's shebang names its interpreter, which the kernel runs with the script's path; #!/usr/bin/env finds it in PATH. Python 3 (python3) is the main scripting language, with the standard library and virtual environments for administration; Perl and others remain common. Java compiles to bytecode run by the JVM: javac, java ClassName, java -jar. Git records snapshots: init, add, commit, status, diff, log, clone, pull, push and branches.
Flashcards
Flashcard
Free version of Lex: generates a lexer (tokeniser) in C
Test your understanding
Question 1 of 5
A script begins #!/usr/bin/python3 but fails on a system where Python lives in /usr/local/bin. Which first line avoids the problem?
Question 2 of 5
When you run ./report and its first line is #!/usr/bin/python3, what does the kernel do?
Question 3 of 5
Which command runs a compiled Java class stored in Hello.class?
Question 4 of 5
After editing a tracked file, git status --short shows M config.sh. Which command records the change in a new commit?
Question 5 of 5
A build fails at the link step because yywrap is undefined. The program's lexer was generated by flex. What is the likely fix?
