Computer Science › Section D · Operating systems › Topic D.6

Operating system security

The operating system oversees everything a computer does, so it is also the computer's main security guard. It decides who may log in, keeps each process inside its own memory, and relies on special processor features so that no ordinary program can seize control of the machine.

Why the OS guards the machine

Every file, block of memory and device passes through the operating system's hands, so it is the natural place to enforce security. Security begins with reliability: if a file manager fault loses part of a file, or a dispatcher bug crashes the system and an hour of work disappears, that data was not safe. Building dependable software is the subject of the section on software engineering. This topic focuses on threats specific to operating systems: outsiders trying to get in, and insiders trying to reach beyond what they are allowed to do.

Accounts and logging in

On a computer used by several people, the OS keeps an account for each authorised user: a record holding the user name, a protected form of the password (normally a hash, not the password itself) and the privileges that user is granted. Each login checks the details given against that record. Accounts are created by the administrator, called the super user or root on Unix-like systems, who can also change system settings, modify critical software and adjust other users' privileges, actions denied to ordinary users. That power is why the administrator's password is the most closely guarded secret on the system.

Passwords and their weaknesses

One of the biggest obstacles to security is careless users. People choose easily guessed passwords such as names, dates and dictionary words, share them, reuse them on many sites and install unapproved software. Attackers' programs try enormous numbers of guesses automatically, so a strong password is one that would take a very long time to find by trying every possibility. Length helps most, since each extra character multiplies the possibilities. Password managers make long, unique passwords practical, and a second factor, such as a code from a phone, protects an account even when a password leaks, as the topic Cybersecurity: encryption and authentication explains.

Worked example: counting the guesses

Assume an attacker with a stolen file of fast, unsalted hashes can test one billion guesses per second. How long to try every password? 1. Eight lowercase letters: each position has 26 choices, so the search space holds 26⁸ ≈ 2.09 × 10¹¹ passwords. 2. At 10⁹ guesses per second that takes about 209 seconds, under 4 minutes. 3. Twelve characters drawn from upper case, lower case and digits: 62 choices per position, so 62¹² ≈ 3.2 × 10²¹ passwords. 4. At the same rate that is about 3.2 × 10¹² seconds, roughly 100,000 years. 5. Check: four extra positions and a bigger character set multiplied the work by about 15 billion. The guessing rate is an assumption; real rates vary widely.

Auditing and sniffing software

Administrators watch for trouble with auditing software, utilities that record and analyse activity. A burst of failed logins suggests password guessing. Activity that does not fit an account's history, such as a user who only writes documents suddenly running system tools, suggests a stolen account. Auditing also hunts for sniffing software, programs left running to record what users do and report it to an intruder. A classic trick is a fake login screen that collects names and passwords. Some Windows setups ask users to press Ctrl+Alt+Delete before signing in because ordinary programs cannot intercept that key combination, so the screen that follows is genuine.

Attacks from within

Once intruders get into an ordinary account, or a legitimate user turns malicious, they try to reach beyond that account's privileges, which is called privilege escalation. With the administrator's password this is easy. Otherwise they must trick the OS: get the memory manager to let a process touch memory outside its area, or the file manager to hand over a protected file. A process that could write anywhere in memory could overwrite the OS itself and take control. So the OS needs the hardware's help to enforce its rules.

Fencing in a process's memory

Processors built for multiprogramming let the OS fence each process in. In the simplest design, two special registers hold the lowest and highest addresses of the process's area. On every memory reference the processor compares the address with these limits; if it falls outside them, the processor triggers an interrupt that hands control back to the OS, which can stop the process. Modern processors do the same job with the memory management unit: each page-table entry also records whether the page may be read, written or executed, and whether user programs may touch it at all.

Privileged mode and privileged instructions

A fence is useless if a process can move it by loading a bigger value into its upper-limit register. So processors have at least two privilege levels: privileged mode, also called kernel mode, where every instruction is allowed, and nonprivileged or user mode, where some are forbidden. The forbidden ones are privileged instructions, such as changing the limit registers, the timer or the privilege level itself. Trying one in user mode causes an interrupt that switches to privileged mode and hands control to the OS. The processor starts in privileged mode; the OS switches to user mode before each time slice, and system calls are the controlled way back in. Intel x86 processors offer four levels; most systems use two.

Worked example: tracing a bounds violation

A process is allotted addresses 40,000 to 49,999, stored by the OS in the two limit registers. 1. The process reads address 45,120: it lies within the limits, so the access goes ahead. 2. It tries to write to address 52,000: that is above 49,999, so the processor raises an interrupt and the OS takes over, typically ending the process. 3. Suppose instead it tries to load 60,000 into its upper-limit register. 4. That is a privileged instruction in user mode, so it is not carried out: an interrupt passes control to the OS. 5. Check: in both attacks the hardware detected the problem and the OS, never the process, decided what happened next.

Least privilege and keeping up to date

Privilege control is complex, and flaws keep being found in real systems. One flaw can be disastrous: a process able to change the timer could keep the processor forever, and one with direct device access could read files without the file manager's checks. Two habits limit the damage. The principle of least privilege gives every user and program only the access it needs, so everyday work runs in a standard account and administrator rights are requested only for a specific task, as sudo does on Linux and macOS. And installing security updates promptly closes flaws that attackers have learned to use.

Common mistakes to avoid

Security is not only about hackers: an OS that crashes or loses files is insecure too. Passwords should not be stored as plain text; systems keep and compare hashes. Complexity rules are not the whole story: a bigger character set raises the base, but every added character multiplies the count again, so a long passphrase can beat a short complex password. Limit registers alone are not enough; without privileged instructions any process could change them. And user mode does not stop a program using the disk or network; it makes the program ask the kernel through system calls, so the OS can check every request.

Key takeaways

The OS secures the machine by being reliable and by controlling access. Accounts, logins and an administrator decide who gets in; long passwords, password managers and second factors defend accounts, and auditing software spots guessing and sniffing. Inside the system, limit registers or page-table permissions fence each process into its own memory, and an out-of-bounds access causes an interrupt. Privileged instructions, allowed only in privileged mode, stop processes from moving those fences or seizing the timer. Least privilege and prompt updates limit the damage when a flaw is found.

Flashcards

Flashcard

Account

OS record of a user's name, password hash and privileges

Checked at every login.

Test your understanding

Question 1 of 5

How many different passwords can be made from exactly 6 lowercase letters (a to z)?

Question 2 of 5

Why can't a process in user mode simply raise its own upper memory limit to get more memory?

Question 3 of 5

A process is allotted addresses 2,000 to 2,999. It tries to read address 3,000. What happens?

Question 4 of 5

Auditing software reports 800 failed login attempts on one account in ten minutes. What does this most likely indicate?

Question 5 of 5

Which practice best follows the principle of least privilege on a family laptop?

← Previous topicNext topic →