vm-obfuscation-detection

name	VM/Obfuscation Detection
description	Detect virtual machines, packers, and code obfuscation — VMProtect, Themida, UPX, control flow flattening
tags	["obfuscation","packer","vmprotect","themida","upx","devirtualization","cff"]
allowed_tools	["detect_obfuscation","get_deobfuscation_advice","analyze_obfuscated_functions","decompile_function","get_disasm","list_imports"]

Task: Virtual Machine and Obfuscation Detection. Identify and analyze code protection and obfuscation.

Detection Goals

1. Identify Known Protectors - VMProtect, Themida, UPX, etc.
2. Detect Obfuscation Patterns - Control flow flattening, opaque predicates
3. Assess Deobfuscation Difficulty - Provide actionable recommendations
4. Document Findings - Create report for team sharing

Mandatory First Steps

1. Run Automatic Detection

   Use `detect_obfuscation` to scan the binary
   This will identify:
   - Known packer/protector signatures
   - Obfuscation patterns
   - High-complexity functions
   

2. Review Detected Protectors

   • Check if protector is known and has unpacking tools
   • Assess difficulty of deobfuscation
   • Research specific techniques for the detected protector

3. Analyze High-Complexity Functions

   Use `analyze_obfuscated_functions` with threshold=200
   This identifies:
   - Functions with excessive basic blocks
   - Dispatcher patterns
   - Junk code indicators
   

Known Protectors

Easy to Unpack

• UPX - upx -d file.exe (one command)
• ASPack - Specialized unpackers available
• PECompact - Can be unpacked manually

Medium Difficulty

• Themida - Dump from memory after unpacking stub
• Enigma - Memory dumping + IAT rebuild
• ASProtect - Specialized tools needed

Hard to Devirtualize

• VMProtect - Requires devirtualization or dynamic analysis
• Tigress - Symbolic execution, partial evaluation
• CodeVirtualizer - VM handler analysis needed
• Denuvo - Very complex, usually not feasible

Obfuscation Patterns

Control Flow Flattening

Indicators:

• Large switch statement with jump table
• Indirect jumps via register
• Dispatcher-based execution
• Spaghetti code structure

Analysis Approach:

1. Identify the dispatcher loop
2. Trace execution flow dynamically
3. Map blocks to original logic
4. Use symbolic execution if needed

Opaque Predicates

Indicators:

• xor eax, eax; test eax, eax (always zero)
• cmp reg, reg (always equal)
• Redundant comparisons

Detection:

• Look for always-true/false conditions
• Find tautological comparisons
• Identify dead code branches

Junk Code Insertion

Indicators:

• Excessive NOPs
• Useless mov instructions
• Push/pop without effect
• Dead stores

Analysis:

• Filter out junk during decompilation
• Focus on meaningful instructions
• Trace data flow ignoring junk

Instruction Substitution

Indicators:

• xor reg, reg vs mov reg, 0
• sub eax, 1 vs dec eax
• Complex sequences for simple operations

Impact:

• Makes static analysis harder
• Increases code size
• May confuse disassemblers

Deobfuscation Strategies

1. Unpacking

Step 1: Run the binary in a debugger
Step 2: Set breakpoint on entry point
Step 3: Let unpacking stub run
Step 4: Dump unpacked code from memory
Step 5: Rebuild import table

2. Devirtualization

Step 1: Identify VM handlers
Step 2: Lift handlers to intermediate representation
Step 3: Symbolically execute VM bytecode
Step 4: Reconstruct original semantics
Step 5: Generate native code

3. Dynamic Analysis

Step 1: Instrument the binary
Step 2: Trace execution path
Step 3: Record actual behavior
Step 4: Map observed behavior to source
Step 5: Identify real logic vs junk

4. Pattern Matching

Step 1: Identify obfuscation patterns
Step 2: Create recognition rules
Step 3: Apply to all functions
Step 4: Simplify/normalize code
Step 5: Repeat until clean

Tools and Techniques

Automatic Tools

• Flare-Emu - Emulation-based analysis
• D810 - Deobfuscation framework
• Unipacker - Universal unpacker
• x64dbg/OllyDbg - Dynamic debugging

Manual Techniques

• Memory Dumping - Extract unpacked code
• API Hooking - Monitor behavior
• Symbolic Execution - angr, Triton
• LLVM Optimization - Simplify obfuscated code

Reporting

For each detected protection:

[Obfuscation] VMProtect v3.x
Type: Virtualization
Difficulty: Very Hard
Coverage: 85% of functions

[Detection]
- Signature: VMProtect strings found
- Sections: .vmp0, .vmp1, .vmp2 detected
- Functions: 85% show dispatcher pattern
- Complexity: Average 500+ instructions per function

[Recommendations]
1. Use VMProtect devirtualization tools (research needed)
2. Focus dynamic analysis on exports/imports
3. Trace execution paths with debugger
4. Consider memory dumping at runtime
5. May require manual reverse engineering

[Workarounds]
- Analyze at import/export boundaries
- Hook interesting APIs directly
- Monitor behavior dynamically
- Skip protected functions initially

Workflow

1. Detection Phase

   • Run detect_obfuscation
   • Identify protector type
   • Assess difficulty

2. Research Phase

   • Run get_deobfuscation_advice for specific protector
   • Research available tools
   • Check community solutions

3. Analysis Phase

   • Use dynamic analysis if static is blocked
   • Focus on unobfuscated wrapper code
   • Hook APIs to understand behavior

4. Reporting Phase

   • Document all findings
   • Share with team via export_analysis
   • Include recommendations

Tips

• Start with imports/exports - often unobfuscated
• Use dynamic tracing to understand behavior
• Don't waste time on heavily protected functions initially
• Focus on what the binary DOES, not HOW it works internally
• Consider specialized tools for specific protectors