Linux › Section Q · Virtualization › Topic Q.3

Runtime-based virtualization

Not every isolation problem needs a virtual machine or a container. Often the clash is inside one programming language: two applications want different versions of the same library. Runtime-based virtualisation, such as Python's virtual environments, solves that for one application at a time.

A third kind of virtualisation

The previous two topics, "Virtual machines" and "Containers: namespaces and cgroups", isolated whole machines and groups of processes. Runtime-based virtualisation is narrower. It gives one application its own private set of language packages while still sharing the system, the kernel and usually the interpreter itself. The word virtual in it is a reminder of the general meaning: an intermediary that makes a shared resource look private. Here the shared resource is a language's library directory, and the intermediary is a small change to where the interpreter looks for packages.

The problem: one language, many users

A distribution uses Python for its own tools, from package-manager helpers to firewall front-ends, and installs the Python libraries those tools need as distribution packages. Now suppose your application needs a newer release of one of those libraries. Upgrading it system-wide with pip could break the distribution's tools; leaving it alone breaks your application. A second application might need yet another version. Because a single library directory can hold only one version of each package, the applications cannot all be satisfied at once. The fix is to give each application a library directory of its own.

Python virtual environments

python3 -m venv creates a virtual environment: a directory with its own bin directory, containing links to the interpreter, its own lib/pythonX.Y/site-packages for packages, and a small pyvenv.cfg file recording which base Python it came from. Running the environment's python, or pip, uses that private site-packages and, by default, ignores packages installed system-wide: pyvenv.cfg says include-system-site-packages = false. Packages installed with the environment's pip go only into the environment. Deleting the directory removes the environment completely, leaving the system untouched. The venv module has been part of Python's standard library since Python 3.3.

Worked example: creating and using an environment

1. Create it: python3 -m venv test-venv. 2. Activate it with the dot command, which runs a script in the current shell: . test-venv/bin/activate. It must be sourced, not executed, because its job is to change the current shell's PATH and set VIRTUAL_ENV, which a child process cannot do for its parent. 3. Check: type python now points inside test-venv, and python -m pip list shows only pip and, on older versions, setuptools. 4. Install what the application needs with python -m pip install. 5. Leave with deactivate. On Windows the same layout uses Scripts instead of bin; a fresh Python 3.11 environment there listed only pip and setuptools.

Activation is optional

Activation is only a convenience for interactive work. All it really does is put the environment's bin directory first in PATH. Scripts, services and cron jobs need not activate anything: they simply run the environment's interpreter by its full path, such as /opt/report/venv/bin/python. A script can also point its shebang there, as in #!/opt/report/venv/bin/python, so it always runs with the right packages. Inside an environment, the interpreter reports sys.prefix as the environment directory, different from sys.base_prefix, which is how Python itself knows it is running inside an environment.

Worked example: a cron job that cannot import

A reporting script works when typed at the terminal but fails from cron with ModuleNotFoundError: No module named 'requests'. 1. At the terminal, the user had activated /opt/report/venv, so python meant the environment's interpreter. 2. cron runs with a minimal environment, so python3 there is the system interpreter, which lacks the package. 3. Confirm: /usr/bin/python3 -c 'import requests' fails, while /opt/report/venv/bin/python -c 'import requests' succeeds. 4. Fix: in the crontab, call /opt/report/venv/bin/python /opt/report/report.py, or set that path in the script's shebang. 5. Never fix it by installing into the system Python with sudo pip.

Distributions protect their Python

To stop accidental damage, Python's PEP 668 lets a distribution mark its interpreter as externally managed. Debian 12, Ubuntu 23.04 and later releases do this, so pip install outside a virtual environment stops with an externally-managed-environment error instead of overwriting packaged files. The intended routes are: install the distribution's package when one exists (python3-requests on both Debian and Fedora), use a virtual environment for application libraries, and use pipx for command-line tools written in Python, since pipx gives each tool its own hidden environment and links its command into ~/.local/bin.

Other languages, same idea

Most modern language ecosystems isolate per project by default. Node.js's npm installs packages into a node_modules directory inside each project, recorded in package.json. Rust's cargo and Go's module system download each project's exact dependency versions, recorded in Cargo.lock or go.sum. Ruby's Bundler pins gems per project in a Gemfile.lock. Version managers go one step further and install whole interpreters side by side, such as pyenv for Python or nvm for Node.js. The Java virtual machine from "Lex, Yacc, scripting languages, and Java" is a process virtual machine of a related kind: it runs bytecode in a managed runtime rather than isolating packages.

Choosing the right tool

Match the isolation to the problem. If two applications need different library versions in the same language, a virtual environment or per-project install is enough, and it costs almost nothing. If they need different system libraries, services or distributions, or must be prevented from seeing each other, use a container. If they need a different kernel, a different operating system or the strongest separation, use a virtual machine. Layers combine: a container image often includes a Python virtual environment, and containers often run inside cloud virtual machines. Lighter isolation starts faster and uses less storage; heavier isolation protects more.

Common mistakes to avoid

Running test-venv/bin/activate as a program instead of sourcing it, so nothing changes. Using sudo pip install to escape the externally-managed error, which can break distribution tools. Forgetting that cron and services do not inherit an activated environment. Copying a virtual environment to another machine or path: its scripts contain absolute paths, so recreate it from a requirements file with pip install -r instead. Committing a virtual environment into Git rather than its requirements list. Expecting a virtual environment to isolate anything but language packages: it is not a security boundary.

Key takeaways

Runtime-based virtualisation gives one application its own language packages while sharing the system and kernel. python3 -m venv dir creates a Python environment with private site-packages; source dir/bin/activate for interactive use, or call dir/bin/python directly from scripts, cron and services; deactivate leaves. PEP 668 makes Debian and Ubuntu refuse system-wide pip installs; use distribution packages, venvs or pipx. npm, cargo, Go modules and Bundler isolate per project. Choose environments, containers or virtual machines by how much must be separated.

Flashcards

Flashcard

Runtime-based virtualisation

Private language packages for one application, sharing the system

Narrower than containers or VMs.

Test your understanding

Question 1 of 5

Why must the activate script be sourced with the dot command rather than run as a program?

Question 2 of 5

A script runs fine at the terminal but fails from cron with ModuleNotFoundError. The package is in /opt/app/venv. What is the fix?

Question 3 of 5

On Debian 12, pip install requests fails with externally-managed-environment. What does this indicate?

Question 4 of 5

Two applications need different versions of the same Python library and nothing else differs. Which isolation is the best fit?

Question 5 of 5

Why should you not copy a virtual environment directory to another server?

← Previous topic