Protection is the mechanism controlling what each running entity may do; security is the policy that decides what should be allowed and the assurance that it holds against adversaries. The OS enforces both on top of hardware support.
Protection vs Security
- Protection: internal, mechanism-focused (how the OS confines a process to its own memory and files).
- Security: broader, adversary-focused (defending against malicious actors, covering authentication, integrity, and confidentiality).
Good protection mechanisms are what make security policies enforceable.
Dual-Mode Operation
The hardware provides at least two privilege levels; the OS builds all isolation on this (see OS Hardware).
- Kernel mode: full instruction set, direct device and MMU access.
- User mode: a safe subset; privileged instructions trap to the kernel.
user process --syscall/trap--> [ kernel mode: validate + act ] --return--> user
privileged instruction in user mode --> hardware trap --> kernel handles/killsCrossing the boundary happens only through defined gates (system calls, interrupts, exceptions), so the kernel mediates every privileged action. x86 actually offers four rings (0 to 3); most systems use only ring 0 and ring 3.
Memory Isolation
The MMU confines each process to its own address space (see Memory Management and Virtual Memory). Per-page permission bits (read, write, execute) enforce finer control:
- W^X (write xor execute): a page is never both writable and executable, blocking classic code injection.
- ASLR: randomize the base of stack, heap, and libraries so an attacker cannot predict addresses.
- Guard pages and stack canaries: detect overflows before they corrupt control data.
Principle of Least Privilege
Every component should hold only the rights it needs, for only as long as it needs them.
- Drop privileges after startup (a server binds port 80 as root, then drops to an unprivileged user).
- Sandbox risky code (browser renderers, containers, seccomp filters that whitelist syscalls).
- Split a monolith into least-privileged services so one compromise is contained.
Access Control Models
Think of a protection matrix: rows are subjects (users, processes), columns are objects (files, devices), cells are allowed operations. It is stored two ways:
| Representation | Stored with | Analogy |
|---|---|---|
| Access Control List | the object | a guest list on the door |
| Capability | the subject | an unforgeable ticket you carry |
UNIX file permissions (owner/group/other x read/write/execute) are a compact ACL. File descriptors are effectively capabilities: holding an open fd grants access without re-checking the path. Access can be discretionary (owner sets policy, DAC) or mandatory (system-wide policy the owner cannot override, MAC, as in SELinux).
Authentication and Authorization
- Authentication: prove identity (something you know, have, or are; passwords, tokens, biometrics). Passwords are stored salted and hashed, never in the clear.
- Authorization: given an identity, decide what it may do (the access-control check above).
Keep the two distinct: authentication answers “who are you,” authorization answers “what may you do.”
Common Threats
| Threat | Mechanism | Defense |
|---|---|---|
| Buffer overflow | overwrite the stack return address | canaries, W^X, ASLR, bounds checking |
| Privilege escalation | trick privileged code into acting for you | least privilege, input validation |
| TOCTOU race | state changes between check and use | atomic checks, operate on fds not paths |
| Malware / trojans | run attacker code with a user’s rights | signing, sandboxing, least privilege |
| Side channels | infer secrets from timing, cache, or power | constant-time code, cache isolation |
| Denial of service | exhaust a shared resource | quotas, rate limits, cgroups |
Confused deputy and TOCTOU
A privileged program (the deputy) can be tricked into misusing its authority on behalf of a caller (the confused-deputy problem). A common instance is a TOCTOU race: a program checks
access(path)thenopen(path), and an attacker swaps the file (via a symlink) in between. The fix is to eliminate the gap: open first, then check the open descriptor, so the check and use refer to the same object.
Isolation Boundaries in Practice
Different mechanisms give different isolation strength at different cost (see Virtualization):
- Process: MMU-enforced address-space separation, weakest but cheapest.
- Container: kernel namespaces and cgroups, shared kernel is the risk.
- Virtual machine: hypervisor boundary, strongest but heaviest.
Defense in Depth
No single mechanism is trusted alone. Layer independent controls (hardware modes, MMU permissions, ACLs, sandboxing, monitoring) so that breaching one does not grant the whole system. Minimize the trusted computing base: the smaller the code that must be correct, the smaller the attack surface.