Linux › Section O · Development tools › Topic O.1

The C compiler and linking

Most of the programs on a Linux system began as C or C++ source code. Knowing how the compiler, the linker and shared libraries fit together explains where those programs come from, and why a program can fail to start with a message about a missing library.

From source code to program

C is a compiled language: the human-readable source code in .c files must be translated into machine code before it runs, as the Computer Science topic "Language implementation: translation and compilers" explains. On Linux the usual compiler is GCC, run as gcc or by the traditional name cc; clang, from the LLVM project, accepts nearly the same options. Many installations leave them out. Install them with sudo apt install build-essential on Debian and Ubuntu, or sudo dnf group install development-tools on current Fedora. The simplest build is gcc -o hello hello.c. Without -o, the result is called a.out. Messages quoted below are GCC's usual wording; details vary slightly between versions.

The four stages

gcc runs a pipeline of separate steps, and options let you stop after any of them. 1. The preprocessor (cpp) handles lines starting with #, such as #include and #define; gcc -E prints its output. 2. The compiler proper turns that into assembly language; gcc -S writes a .s file. 3. The assembler produces an object file of machine code; gcc -c writes a .o file. 4. The linker (ld) joins object files and libraries into an executable, resolving each name one file uses and another defines. You rarely run ld yourself: gcc calls it with the right system files when you give it .o files.

Worked example: a two-file program

A program averages numbers. mean.h declares double mean(const double *v, int n); mean.c defines that function, and main.c calls it on the array {2.0, 4.0, 9.0} and prints the result with %.2f. 1. Compile each file to an object: gcc -c mean.c and gcc -c main.c give mean.o and main.o. 2. Link them: gcc -o avg main.o mean.o. 3. Run ./avg; it should print 5.00, since (2 + 4 + 9) / 3 = 5. 4. Now edit only mean.c. Only mean.c needs recompiling, followed by the link step; main.o is reused. 5. With dozens of files, tracking which ones changed by hand is error-prone, which is exactly the job of the next topic, "make".

Header files and the preprocessor

A header file holds declarations shared between source files. #include <stdio.h> searches the system include directories, chiefly /usr/include; #include "mean.h" searches the source file's own directory first. If a header is elsewhere, add its directory with -I, as in gcc -c -I/opt/geo/include map.c. A missing header stops compilation with an error such as map.c:1:10: fatal error: geo.h: No such file or directory. On a distribution system, the fix is usually installing the library's development package, named like libpng-dev on Debian or libpng-devel on Fedora. -DNAME=value defines a macro, and #ifdef DEBUG ... #endif includes code only when DEBUG is defined.

Linking against libraries

A library is a bundle of precompiled object code, such as the maths library libm. Name it with -l, dropping the lib prefix and suffix: -lm links libm. If it lives outside the standard directories, add -L with its directory: gcc -o map map.o -L/opt/geo/lib -lgeo. Put libraries after the object files that need them, because the GNU linker resolves names in command-line order. Forget a library and the link fails with an error that ends undefined reference to 'sqrt' or similar. The same error appears when one of your own object files is missing, so first ask whose function it is. nm --defined-only on a library lists the names it provides.

Static and shared libraries

A static library (.a) is copied into the executable at link time. The program then runs without the library file, but every program carries its own copy, and a security fix to the library helps nobody until each program is relinked. A shared library (.so, for shared object, as in libc.so.6) is only referenced at link time and loaded when the program starts. One copy on disk and in memory serves every program, and updating the library fixes them all at once. That is why, after a library update, you may be told to restart services or reboot: running programs still hold the old version. The cost is that the library must be found at run time.

ldd and the runtime linker

An executable records only the names of the shared libraries it needs. When it starts, the runtime linker, ld.so (on x86-64, /lib64/ld-linux-x86-64.so.2), finds and loads them. ldd prints the mapping: each needed name, an arrow, and the file found, such as libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (illustrative). ld.so(8) gives the search order: an old-style rpath built into the binary (ignored if a runpath is present), then LD_LIBRARY_PATH, then a newer-style runpath, then the cache /etc/ld.so.cache, then /lib and /usr/lib. The cache lists libraries in directories named in /etc/ld.so.conf and /etc/ld.so.conf.d; after changing them, run ldconfig as root. Never run ldd on an untrusted program, because it may execute it; objdump -p prog | grep NEEDED is the safe check.

Building in a search path

Suppose a program needs libgeo.so.1 from /opt/geo/lib, a directory that should not go into the system-wide cache. Tell the linker to record the path inside the executable: gcc -o map map.o -L/opt/geo/lib -Wl,-rpath,/opt/geo/lib -lgeo. -Wl passes an option through gcc to the linker; -L is still needed so the linker can find the library at build time. readelf -d map then shows an RPATH or RUNPATH entry, depending on how the toolchain is configured. To change the path in an existing binary, patchelf --set-rpath can rewrite it, though setting it at build time is cleaner.

Worked example: a program that will not start

./map fails with: error while loading shared libraries: libgeo.so.1: cannot open shared object file: No such file or directory. 1. Read the message: the build worked, but ld.so cannot find a library at run time. 2. Confirm with ldd ./map (it is your own program, so ldd is safe): libgeo.so.1 => not found. 3. Locate the file, say /opt/geo/lib/libgeo.so.1. 4. Best fix: relink with -Wl,-rpath,/opt/geo/lib. 5. If many programs need it, add the directory to a file in /etc/ld.so.conf.d and run ldconfig. 6. For a closed-source program you cannot relink, use a wrapper script: set LD_LIBRARY_PATH=/opt/geo/lib, export it, then exec the real program with "$@". The variable then reaches only that program, not the user's other commands.

Common mistakes to avoid

Setting LD_LIBRARY_PATH in a startup file or for builds: every program then searches those directories first, which is slow and can load the wrong version of a library. Putting -lm before the object files that use it. Assuming every undefined reference means a missing library, when an object file may be missing. Adding odd directories to /etc/ld.so.conf.d instead of using an rpath. Forgetting ldconfig after changing that configuration. Running ldd on a program you do not trust. Installing a library package without its -dev or -devel package and then wondering why its header cannot be found.

Key takeaways

gcc (or clang) preprocesses, compiles, assembles and links; -E, -S and -c stop after each of the first three, and -o names the output. Compile files separately with -c, then link the objects. -I adds header directories, -L library directories, and -l names a library, placed after the objects. Static .a libraries are copied in; shared .so libraries are loaded at run time by ld.so, using rpath, LD_LIBRARY_PATH, runpath, /etc/ld.so.cache and the default directories. ldd, ldconfig, readelf -d and -Wl,-rpath diagnose and fix library problems.

Flashcards

Flashcard

gcc -o hello hello.c

Compile and link hello.c into the program hello

Without -o the output is a.out.

Test your understanding

Question 1 of 5

A link step fails with an undefined reference to a function named area, which you wrote in shapes.c. What is the most likely cause?

Question 2 of 5

Which command compiles mean.c into an object file without linking?

Question 3 of 5

A program starts with: error while loading shared libraries: libgeo.so.1: cannot open shared object file. What does this show?

Question 4 of 5

Why is a security fix to a shared library more effective than the same fix to a static library?

Question 5 of 5

A program needs a library in /opt/geo/lib that should not be added system-wide. What is the recommended fix?

← Previous topicNext topic →