custom-memory-heap-crash

name

custom-memory-heap-crash

description

Guide for debugging crashes related to custom memory heaps, particularly use-after-free issues caused by static destruction ordering, DEBUG vs RELEASE discrepancies, and custom allocator lifecycle problems. This skill should be used when investigating crashes that occur only in RELEASE builds, memory-related crashes involving custom allocators, static initialization/destruction order issues, or use-after-free bugs in C++ applications.

Custom Memory Heap Crash Debugging

This skill provides systematic approaches for debugging crashes related to custom memory heaps, with emphasis on static destruction ordering issues, DEBUG vs RELEASE discrepancies, and memory lifecycle problems.

Problem Recognition

Apply this skill when encountering:

• Crashes that occur only in RELEASE builds but not DEBUG builds
• Segmentation faults or access violations during program shutdown
• Use-after-free errors involving custom allocators
• Crashes in standard library code (locale, iostream, etc.) when custom heaps are involved
• Memory lifecycle issues during static destruction phase

Investigation Approach

Phase 1: Reproduce and Characterize

1. Build both configurations: Compile the application in both DEBUG and RELEASE modes to confirm the discrepancy
2. Identify crash timing: Determine if the crash occurs during:
   • Normal execution
   • Program shutdown (static destruction phase)
   • Library initialization
3. Collect crash information: Use GDB or equivalent debugger to obtain:
   • Full backtrace at crash point
   • Register values and memory state
   • The specific instruction causing the crash

Phase 2: Understand Memory Lifecycle

1. Map allocator lifecycle: Document when custom heaps are:
   • Created (constructor timing)
   • Active (during main execution)
   • Destroyed (destructor timing)
2. Identify static objects: List all static/global objects that may allocate memory
3. Trace allocation sources: Determine which allocator (custom vs system) handles each allocation

Phase 3: Analyze Static Destruction Order

1. Review destruction sequence: Static objects are destroyed in reverse order of construction
2. Check cross-dependencies: Identify objects that depend on the custom heap but may be destroyed after it
3. Examine library internals: Standard library components (locales, facets, streams) may register objects that outlive custom heaps

Common Root Causes

Use-After-Free During Static Destruction

Pattern: Objects allocated from a custom heap are accessed after the heap is destroyed.

Symptoms:

• Crash in destructor or cleanup code
• Backtrace shows standard library cleanup (e.g., locale facet destruction)
• Crash accesses memory that was valid earlier

Investigation:

• Check if standard library objects (locale facets, stream buffers) are allocated from the custom heap
• Verify the custom heap outlives all its allocations

DEBUG vs RELEASE Allocation Differences

Pattern: Different allocation patterns between build configurations cause memory to come from different sources.

Symptoms:

• Works in DEBUG, crashes in RELEASE
• Memory addresses differ between configurations
• Conditional compilation affects allocator selection

Investigation:

• Examine preprocessor conditionals affecting memory allocation
• Check if DEBUG mode uses system allocator while RELEASE uses custom heap
• Look for #ifdef DEBUG or #ifndef NDEBUG blocks around allocation code

Library-Internal Allocations

Pattern: Standard library internally allocates memory that gets routed through custom allocators.

Symptoms:

• Crash during library cleanup code
• Backtrace shows internal library functions
• No obvious user code involvement

Investigation:

• Trace which library operations trigger allocations
• Check if locale, iostream, or other library initializations use the custom heap
• Examine library source code if available

Solution Strategies

Strategy 1: Force Early Initialization

Trigger library initialization before the custom heap is created, ensuring library-internal allocations use the system allocator.

Implementation approach:

• Call library functions in user_init() or before heap creation
• For locale issues: instantiate std::locale() early
• For iostream issues: perform I/O operations early

Strategy 2: Extend Heap Lifetime

Ensure the custom heap outlives all objects allocated from it.

Implementation approach:

• Use static local variables with guaranteed destruction order
• Implement explicit cleanup before heap destruction
• Consider lazy destruction or leaking the heap intentionally

Strategy 3: Exclude Library Allocations

Prevent library-internal allocations from using the custom heap.

Implementation approach:

• Modify allocator selection logic to exclude certain allocation types
• Use thread-local flags during library initialization
• Implement allocation source tracking

Verification Checklist

After implementing a fix:

1. Build verification: Confirm both DEBUG and RELEASE builds compile without errors
2. Runtime verification: Run both configurations without crashes
3. Memory leak check: Use Valgrind or equivalent to verify no memory leaks introduced
4. Stress testing: Run multiple iterations to catch intermittent issues
5. Destruction order verification: Confirm proper cleanup sequence with logging if needed

Debugging Tools and Techniques

GDB Commands for Memory Issues

# Get backtrace at crash
bt full

# Examine memory at address
x/16xg <address>

# Check if address is valid
info proc mappings

# Set breakpoint on destructor
break ClassName::~ClassName

# Watch memory location
watch *<address>

Valgrind Usage

# Basic memory check
valgrind --leak-check=full ./program

# Track origins of uninitialized values
valgrind --track-origins=yes ./program

# Detect invalid reads/writes
valgrind --read-var-info=yes ./program

Common Pitfalls

1. Incomplete initialization: Triggering partial library initialization may not allocate all necessary objects. Verify the specific code path that causes problematic allocations.

2. Multiple initialization points: Library components may be initialized from multiple code paths. Ensure all paths are covered.

3. Thread safety assumptions: Static initialization may involve thread-safety mechanisms that interact with custom allocators.

4. Optimization effects: Compiler optimizations may reorder or eliminate code that affects allocation timing.

5. GDB command syntax: When debugging, use proper quoting and escaping. Test commands interactively before scripting.

6. Assuming single root cause: Multiple allocation sources may contribute to the problem. Verify each fix addresses all crash scenarios.

Decision Framework

When investigating, follow this systematic approach:

1. Can the crash be reproduced reliably? If not, add logging to capture crash state.
2. Is this a DEBUG vs RELEASE discrepancy? Check preprocessor conditionals.
3. Does the crash occur during shutdown? Focus on static destruction order.
4. Is library code involved? Investigate library initialization and cleanup.
5. Is memory being accessed after free? Trace the allocation source and lifetime.

Apply fixes incrementally and verify each change before proceeding.