osdev-security

name	osdev-security
description	OS-level security mechanisms. SMEP/SMAP, NX/XD bit, stack canaries, KASLR, IOMMU, ring isolation, and KPTI. Use when hardening a kernel, implementing memory protection, adding stack guards, or configuring CPU security features.
origin	MCC

OS Security Mechanisms

Reference for implementing kernel security features: page-level protections, supervisor mode hardening, stack protection, address space randomization, kernel page table isolation, and DMA protection.

When to Use

• Enabling NX/XD bit and enforcing W^X policy in your page tables
• Turning on SMEP and SMAP to isolate kernel from user memory
• Adding stack canaries to detect buffer overflows
• Implementing KASLR to randomize kernel load address
• Setting up KPTI to mitigate Meltdown-class attacks
• Configuring IOMMU to prevent DMA attacks from devices
• Validating user pointers in syscall handlers
• Auditing your kernel for missing security hardening

Page-Level Protection

NX/XD Bit

The No-Execute bit (bit 63 of a page table entry on x86-64) marks a page as non-executable. Any attempt to execute code from an NX-marked page triggers a page fault.

Why it matters: Without NX, a buffer overflow that writes shellcode into a data buffer can jump to that buffer and execute it. NX makes data pages non-executable, breaking this attack chain. It is the single most important page-level security feature.

Enabling NX on x86-64:

1. Check CPUID leaf 0x80000001, EDX bit 20 (NX support)
2. Set IA32_EFER.NXE (bit 11) via WRMSR
3. Set bit 63 in PTEs for all data pages

// Enable NX
uint64_t efer = rdmsr(0xC0000080);
efer |= (1 << 11);  // NXE
wrmsr(0xC0000080, efer);

// In page table entry:
#define PTE_NX  ((uint64_t)1 << 63)
// Data pages:  PTE_PRESENT | PTE_WRITABLE | PTE_NX
// Code pages:  PTE_PRESENT  (no NX, no writable)

Critical mistake: Using bit 63 without enabling EFER.NXE causes a reserved-bit page fault. The CPU treats bit 63 as reserved when NXE=0.

On ARM, the equivalent bits are PXN (Privileged Execute Never) and UXN (User Execute Never) in the PTE. On RISC-V, execute permission is controlled by the X bit in the PTE (pages are non-executable by default unless X is set).

W^X Policy

Every page should be either Writable OR Executable, never both. This is called W^X (write XOR execute).

Page Type	Flags	Why
Code (.text)	R + X	Code must execute but should never be written to at runtime
Data (.data, .bss, heap)	R + W + NX	Data needs writes but must never execute
Read-only data (.rodata)	R + NX	Constants need neither writes nor execution
Stack	R + W + NX	Stack is pure data; executing from it is always an exploit

Enforcement: Audit your page table setup code to ensure no mapping has both PTE_WRITABLE and executable (no NX) set simultaneously. JIT compilers are the one legitimate exception -- they must map pages as RW, write code, then remap as RX.

Supervisor Mode Protections

SMEP (Supervisor Mode Execution Prevention)

SMEP prevents the kernel (ring 0) from executing code mapped in user-space pages. Enable by setting CR4 bit 20.

Why it matters: Without SMEP, a kernel exploit that hijacks a function pointer can point it at user-mapped memory containing attacker-controlled code. This is called a "ret2user" attack. SMEP forces the CPU to fault instead of executing user pages in kernel mode.

Enable:

// Check CPUID leaf 7, ECX=0, EBX bit 7
if (cpu_has_feature(7, 0, 'b', 7)) {
    uint64_t cr4 = read_cr4();
    cr4 |= (1 << 20);  // SMEP
    write_cr4(cr4);
}

On ARM, this protection is automatic via PXN (Privileged Execute Never) bits in page table entries. Kernel page tables should set PXN on all user-accessible pages.

SMAP (Supervisor Mode Access Prevention)

SMAP prevents the kernel from reading or writing user-space memory at all. Enable by setting CR4 bit 21.

Why it matters: Without SMAP, kernel code can accidentally (or maliciously through exploitation) dereference user-controlled pointers. SMAP ensures that any kernel access to user memory faults unless explicitly permitted.

Intentional access: When the kernel legitimately needs to access user memory (e.g., copy_from_user), temporarily disable SMAP with STAC (Set AC flag) and re-enable with CLAC (Clear AC flag):

// Check CPUID leaf 7, ECX=0, EBX bit 20
if (cpu_has_feature(7, 0, 'b', 20)) {
    uint64_t cr4 = read_cr4();
    cr4 |= (1 << 21);  // SMAP
    write_cr4(cr4);
}

// Safe user memory access pattern:
static inline long copy_from_user(void *dst, const void __user *src, size_t n) {
    if (!validate_user_ptr(src, n))
        return -EFAULT;

    stac();  // Temporarily allow user memory access
    // Fault handler must catch page faults here and return -EFAULT
    memcpy(dst, src, n);
    clac();  // Re-enable SMAP protection
    return 0;
}

Critical: Keep STAC/CLAC windows as small as possible. Never leave SMAP disabled across function calls or after error paths.

See references/cpu-security-features.md for complete SMEP/SMAP/NX enable code and copy_from_user implementation.

Stack Protection

Stack Canaries

A stack canary is a known value placed between local variables and the saved return address. Before a function returns, the compiler-generated code checks that the canary has not been overwritten. If it has, a buffer overflow has occurred and the system halts rather than executing attacker-controlled code.

Kernel implementation requirements:

1. Define the __stack_chk_guard symbol (the canary value)
2. Implement __stack_chk_fail (called when canary mismatch is detected)
3. Compile with -fstack-protector-all (protects all functions) or -fstack-protector-strong (protects functions with arrays/address-taken locals)

// Global canary value -- randomize at boot
uintptr_t __stack_chk_guard;

void init_stack_canary(void) {
    // Use hardware RNG if available (RDRAND on x86)
    if (cpu_has_rdrand()) {
        __stack_chk_guard = rdrand64();
    } else {
        // Fallback: use address of a stack variable XORed with timer
        // Not cryptographically strong but better than a constant
        uintptr_t val;
        __stack_chk_guard = (uintptr_t)&val ^ rdtsc();
    }
    // Ensure at least one byte is zero (catches string overflows)
    __stack_chk_guard &= ~(uintptr_t)0xFF;
}

__attribute__((noreturn))
void __stack_chk_fail(void) {
    panic("stack smashing detected");
}

Guard Pages

Place an unmapped (not-present) page at the bottom of each kernel stack. If a stack overflow grows past the stack allocation, it hits the guard page and triggers a page fault instead of silently corrupting adjacent memory.

+-------------------+  <- stack top (highest address)
|    Stack space     |
|        ...         |
|   (grows down)     |
+-------------------+  <- stack bottom
|   GUARD PAGE       |  <- unmapped, triggers #PF on access
+-------------------+

Why both canaries and guard pages: Canaries detect overwrites of the return address (small buffer overflows). Guard pages detect the stack growing beyond its allocation (large overflows or infinite recursion). You need both.

KASLR (Kernel Address Space Layout Randomization)

KASLR loads the kernel at a random base address on each boot, making it harder for exploits to know where kernel code and data structures are in memory.

Implementation:

1. Entropy source: Use RDRAND/RDSEED (x86), or timer jitter as fallback
2. Relocatable kernel: Compile with -fPIC or use a relocatable linker script. All kernel addresses must be position-independent or patched at boot
3. Randomization granularity: Typical slide is a multiple of 2MB (huge page alignment) within a range (e.g., 1GB window)
4. Apply at boot: Choose random offset, relocate kernel, then continue boot

Without KASLR:  Kernel always at 0xFFFFFFFF80000000
With KASLR:     Kernel at 0xFFFFFFFF80000000 + random_offset
                random_offset = align_2mb(random() % KASLR_RANGE)

Impact on debugging: With KASLR enabled, stack traces show randomized addresses. You need to communicate the KASLR offset to the debugger. Log it at boot or provide a debug interface to retrieve it.

Limitations: KASLR is not a security boundary -- it is a mitigation that raises the bar. Information leaks (reading kernel pointers from /proc, timing side channels) can defeat KASLR. It is most effective combined with other protections (SMEP, SMAP, NX).

KPTI (Kernel Page Table Isolation)

KPTI maintains separate page tables for user mode and kernel mode. In user mode, kernel pages are not mapped at all (except for a tiny trampoline). This mitigates Meltdown-class attacks where speculative execution can leak kernel memory to user space.

How it works:

• User page tables: Map user space normally. Map only a small kernel trampoline page (containing the syscall/interrupt entry code that switches CR3)
• Kernel page tables: Map everything (kernel + user space)
• On syscall/interrupt entry: The trampoline code switches CR3 from user page tables to kernel page tables
• On return to user space: Switch CR3 back to user page tables

Syscall entry path:
  1. SYSCALL lands on trampoline (mapped in both page tables)
  2. Trampoline: mov cr3, kernel_page_table
  3. Jump to real syscall handler
  4. ... handle syscall ...
  5. mov cr3, user_page_table
  6. SYSRET back to user space

Performance cost: Every kernel entry/exit requires a CR3 switch, which flushes the TLB (unless PCID is used). With PCID (Process Context ID), the CPU can tag TLB entries and avoid full flushes, reducing the overhead from ~30% to ~5%.

When to implement: KPTI is critical on Intel CPUs affected by Meltdown (most pre-2019 Intel CPUs). AMD CPUs are generally not affected and can skip KPTI. Check vendor ID from CPUID leaf 0.

IOMMU

Why IOMMU Matters

Without an IOMMU, any PCI/PCIe device with DMA capability can read and write arbitrary physical memory. A malicious device (or a compromised device via a malicious driver) can:

• Read kernel memory (steal encryption keys, credentials)
• Write kernel memory (inject code, modify page tables)
• Bypass all CPU-level protections (SMEP, SMAP, NX are CPU-only)

The IOMMU sits between devices and physical memory, translating device DMA addresses through its own set of page tables. Only memory explicitly mapped for a device is accessible to that device.

Intel VT-d

Intel's IOMMU implementation:

• Hardware discovers IOMMU via ACPI DMAR (DMA Remapping) table
• Each IOMMU manages a set of PCI devices
• The IOMMU maintains per-device page tables (similar format to CPU page tables)
• The OS programs these page tables to restrict which physical pages each device can access

AMD-Vi

AMD's equivalent to VT-d:

• Discovered via ACPI IVRS (I/O Virtualization Reporting Structure) table
• Similar concept: per-device DMA page tables
• Additionally supports device-level interrupt remapping

Minimal implementation: For a hobby kernel, start by enabling the IOMMU in passthrough mode (all DMA allowed), then progressively restrict devices as you add drivers. The critical first step is parsing the ACPI table to find the IOMMU and initializing its base registers.

See references/iommu.md for VT-d architecture details, DMAR table parsing, and IOMMU page table setup.

Syscall Security

Every syscall handler must treat all arguments as untrusted. User space controls register values and can pass any pointer, any size, any value.

Validate User Pointers

Before dereferencing any pointer from user space:

1. Range check: Verify the pointer falls within user address space (below the kernel/user split)
2. Size check: Verify ptr + size does not overflow and stays within user space
3. Access check: Verify the page is mapped and accessible (let the page fault handler catch unmapped pages)

#define USER_SPACE_END  0x00007FFFFFFFFFFF  // x86-64

static inline bool validate_user_ptr(const void *ptr, size_t size) {
    uintptr_t addr = (uintptr_t)ptr;

    // Check for overflow
    if (addr + size < addr)
        return false;

    // Check within user address space
    if (addr + size > USER_SPACE_END)
        return false;

    return true;
}

copy_from_user / copy_to_user Pattern

Never dereference user pointers directly. Always use dedicated functions that:

1. Validate the address range
2. Temporarily permit user memory access (STAC on x86 with SMAP)
3. Handle page faults gracefully (return error, do not panic)
4. Re-enable protection (CLAC)

Fault handling: Register the copy region with the exception table. If a page fault occurs during the copy, the fault handler checks the exception table and redirects execution to an error return path instead of panicking.

Capability Checks

Before performing any privileged operation in a syscall:

• File operations: Check file permissions against the calling process's UID/GID
• Process operations: Verify the caller has permission to signal/ptrace the target
• System operations: Check for root/CAP_SYS_ADMIN before allowing raw I/O, module loading, or clock changes
• Memory operations: Validate mmap flags, prevent mapping kernel addresses

Security Checklist

Before considering your kernel hardened:

• NX enabled: IA32_EFER.NXE set, all data/stack pages have NX bit
• W^X enforced: No page is both writable and executable
• SMEP enabled: CR4.SMEP set (if CPUID reports support)
• SMAP enabled: CR4.SMAP set, user access only via STAC/CLAC windows
• Stack canaries: __stack_chk_guard randomized at boot, -fstack-protector-strong in CFLAGS
• Guard pages: Unmapped page at bottom of every kernel stack
• CR0.WP set: Kernel respects read-only page mappings
• User pointers validated: Every syscall checks pointer range before access
• copy_from/to_user used: No raw dereference of user pointers in kernel code
• KASLR: Kernel loaded at random base address (if targeting production use)
• KPTI: Separate user/kernel page tables on Meltdown-affected CPUs
• IOMMU: DMA from devices restricted to explicitly mapped regions
• Integer overflow checks: Syscall size/offset arguments checked for overflow before use
• Null pointer guard: Virtual address 0 is never mapped (prevents null deref exploits)

Related Skills

• osdev-architecture -- CR4, EFER, CPUID feature detection for security features
• osdev-paging -- PTE flags, NX bit, page table structure
• osdev-interrupts -- exception handlers for page faults and #GP
• osdev-process-scheduling -- user/kernel mode transitions, syscall paths
• osdev-boot-sequence -- early security feature initialization