Day 2 - Source Compilation and Libraries: What tar→configure→make→make install Really Mean

📌 Key Takeaways

• The 4 steps of a source install each have a clear role — tar (unpack) → ./configure (survey the system + generate a Makefile) → make (compile and link) → make install (copy to system paths).
• --prefix sets the top-level installation path (default /usr/local). To avoid conflicts with packages, use /usr/local or /opt.
• A source install is not recorded in the rpm/dpkg database. You cannot remove it with yum remove; only make uninstall or manual deletion works — the biggest drawback of source installs.
• Libraries split into static (.a, included in the executable at compile time) and dynamic (.so, loaded and shared at run time).
• Use ldd to see dependencies, and tell the system about a new .so path by registering it in /etc/ld.so.conf (.d) + running ldconfig. This is the standard fix for the "cannot open shared object file" error.

Unpacking — Extracting a Source Bundle with tar

• Source code is usually distributed as a compressed archive of the form name-version.tar.gz (or .tar.bz2, .tar.xz).
• Why compile from source: when you need a newer version that isn't in the repository, want to choose build options yourself, or want a binary optimized for your own CPU.

tar -zxvf nginx-1.24.0.tar.gz    # gzip decompress (z) + extract (x) + verbose (v) + file (f)
tar -jxvf nginx-1.24.0.tar.bz2   # bzip2 compression (j)
tar -Jxvf nginx-1.24.0.tar.xz    # xz compression (uppercase J)
cd nginx-1.24.0                   # enter the unpacked directory

The f option of tar must come last, because it reads the argument immediately after it as the "archive file name". In tar -zxvf file.tar.gz, the file.tar.gz after f is the target. If you change the order as in -zxfv, it mistakes v for the file name and errors out. Modern tar auto-detects the compression type (z/j/J) when extracting even if you omit it, but on the exam, the choice that states it explicitly is treated as the answer.

The unpacked directory almost always contains README, INSTALL, a configure script, Makefile.in, and so on. Reading the INSTALL file before installing is the standard practice — the required build tools or libraries and any unusual options are written there. You can also use tar -tzvf file.tar.gz to check only the list of contents before extracting (t = list).

./configure — Surveying the System and Generating the Makefile

• ./configure is a shell script included in the source directory that examines your system environment.
• What it checks: which compilers exist, whether the required libraries and headers are installed, and what the CPU architecture is.
• Based on the results, it generates a Makefile suited to this system.

./configure                              # check with default settings + generate Makefile
./configure --prefix=/usr/local/nginx    # specify the installation location
./configure --with-http_ssl_module       # include the SSL module (enable a feature)
./configure --without-http_rewrite_module  # leave out a specific feature
./configure --help                       # show all available options

--prefix sets the top-level path to which make install will later copy files. The default is usually /usr/local. With --prefix=/opt/myapp, executables go to /opt/myapp/bin and libraries to /opt/myapp/lib. It is the key option when you want to isolate a source install in a single directory. To avoid conflicts with the package manager, people usually use /usr/local or /opt.

⚠️ Trap: It is very common for ./configure to stop saying "required library is missing (configure: error: ... not found)". This happens because the development header package (with the -devel or -dev suffix) is missing. For example, to build SSL support you need openssl-devel (RHEL) or libssl-dev (Debian). Most compile errors occur at this stage, from missing dependent development packages.

Not every source tree has a configure. This script is generated by GNU Autotools (autoconf/automake), and is itself shipped inside the source distribution. Projects using CMake use cmake . instead, and Python uses setup.py. But the traditional flow that Linux Master asks about is the Autotools-based ./configure.

make — Compile and Link

• make compiles the source files as instructed by the Makefile that configure generated.
• It then links the resulting object files with libraries to produce an executable.
• This is the step where the actual "translation" happens.

make            # compile + link according to the Makefile
make -j4        # parallel compilation on 4 CPU cores (faster)
make clean      # remove build products (before rebuilding)

Compilation is actually two sub-steps. (1) Compile: translate each .c source into an .o (object file, a fragment of machine code). (2) Link: bundle multiple .o files and libraries (.so/.a) into a single executable. In the make output, if you tell apart gcc -c ... (compile) from the final gcc -o executable ... (link), you can see this flow. An "undefined reference" error almost always occurs when a library can't be found during the link step.

make compares file modification times and recompiles only the parts that changed. So in a large project, after changing a single line of code and running make again, only that file and the parts depending on it are built, not everything, which makes it fast. To rebuild cleanly from scratch, run make clean and then make. When run without arguments, make executes the first target in the Makefile (usually all).

make install — Copying to System Paths

• The final make install copies the finished executables, libraries, and documentation to the system paths specified by --prefix.
• Usually only this step needs root privileges (because it writes to system areas such as /usr/local).

sudo make install    # copy build products to the system paths (install)

⚠️ Trap: A program installed by compiling from source is not recorded in the package database (rpm/dpkg). Therefore it does not appear in rpm -qa or dpkg -l, and cannot be removed with yum remove. To remove it, use make uninstall (if the Makefile supports it) or delete the installed files manually. This "management blind spot" is the biggest drawback of source installs.

# Try it yourself — a typical source install flow (assuming development tools are installed)
tar -zxvf hello-2.12.tar.gz
cd hello-2.12
./configure --prefix=/usr/local/hello
make
sudo make install
/usr/local/hello/bin/hello       # run the installed program

Static Libraries vs Dynamic Libraries

• Code that many programs use in common (e.g. math functions, encryption) is bundled into libraries and reused.
• They split into two kinds depending on the linking method.

With a static library (.a), that code is copied into the executable at compile time. So the executable is large but runs on its own. With a dynamic library (.so, shared object), the executable keeps only a reference saying "this library is needed", and the dynamic linker loads it into memory the moment it runs. So the executable is small, and multiple programs share the same .so in memory. Another advantage of dynamic libraries is that when a security patch comes out, replacing a single .so takes effect for all programs.

A dynamic library usually consists of an actual file of the form libNAME.so.X.Y.Z, plus symbolic links pointing to it: libNAME.so.X (the soname) and libNAME.so (for development). Thanks to the version numbering scheme, a library can be replaced within a compatible range. Executables record their dependencies by soname (e.g. libssl.so.1.1).

ldd · ldconfig · /etc/ld.so.conf — Handling the Dynamic Linker

• Since dynamic libraries are "loaded at run time", management tools need to answer two questions.
  • Which .so files does an executable need? → ldd
  • Where will those .so files be found? → /etc/ld.so.conf + ldconfig

ldd /usr/bin/nginx        # list of dynamic libraries this executable depends on
ldconfig                   # regenerate the library cache (/etc/ld.so.cache)
ldconfig -p                # list of libraries currently registered in the cache

# Registering a new library path with the dynamic linker
echo "/usr/local/mylib/lib" > /etc/ld.so.conf.d/mylib.conf
ldconfig                   # refresh the cache → now the .so files in that path can be found

ldd (list dynamic dependencies) shows the dynamic libraries an executable depends on and where they actually link to. If not found appears, the system can't find that library and the program fails to run. In that case you must add the library path to /etc/ld.so.conf (or /etc/ld.so.conf.d/*.conf) and refresh the cache with ldconfig.

Instead of searching directories every time, the dynamic linker consults a binary cache called /etc/ld.so.cache to find libraries quickly. ldconfig scans /lib, /usr/lib, and the paths listed in /etc/ld.so.conf to rebuild this cache. So if you installed a new .so outside the system paths, you must run ldconfig for it to be recognized. You can temporarily add a path with the environment variable LD_LIBRARY_PATH, but for permanent registration, ld.so.conf + ldconfig is the standard.

It is common for a program installed from source to fail to run with "error while loading shared libraries: libxxx.so.1: cannot open shared object file". The cause is that the .so is in a non-standard path such as /usr/local/lib that the dynamic linker doesn't know about. The fix: add that path to ld.so.conf.d and run ldconfig. This scenario is one you meet on both the exam and in real work.

Early Unix used only static linking. In the days when memory was precious, copying the same library code redundantly into every executable was a big waste. When dynamic shared libraries (.so) were introduced in the late 1980s, they saved disk and memory dramatically, and opened the way to apply a security patch to the whole system by replacing just one library. The response to Heartbleed (the OpenSSL vulnerability) in 2014 was fast for the same reason: swapping out just libssl.so instantly patched every program dynamically linked to it. It is a classic example of the value of dynamic linking.

📖 Glossary

• Source compilation : Translating human-written C/C++ code into machine code on your own system and installing it. The opposite of a package install, where you receive a pre-built bundle.
• ./configure : A script that surveys the system, checks the compiler, libraries, and architecture, and creates a Makefile suited to that environment.
• Makefile : A build instruction file that lists what to compile and link in what order. make reads it.
• --prefix : The top-level path to which make install copies files. The default is usually /usr/local.
• -devel / -dev package : A collection of header files needed for compilation. Configure stopping because these are missing is the most common case.
• Compile vs Link : The step that translates .c into .o machine-code fragments / the step that bundles those fragments with libraries to finish the executable.
• undefined reference : The error that occurs when a required library or function cannot be found during the link step.
• Static library (.a) : A method where the code is copied into the executable at compile time. The file grows larger but runs on its own.
• Dynamic library (.so) : A method where it is loaded the moment it runs and shared in memory by multiple programs. A patch is reflected everywhere just by replacing it.
• soname : A library name indicating a compatible version, such as libssl.so.1.1. Executables record their dependencies by this name.
• ldd : A command that shows which dynamic libraries an executable depends on and where they actually link to.
• ldconfig / /etc/ld.so.cache : A command that scans library paths and rebuilds the cache / the binary cache the dynamic linker consults.