Advanced reverse engineering is the discipline of analyzing obfuscated, packed, or firmware binaries using automated program analysis: symbolic execution (angr, KLEE, manticore) for path exploration, decompiler confusion techniques (Hex-Rays deobfuscation, Ghidra script automation), binary diffing (BinDiff, Diaphora, Kam1n0) for variant analysis, firmware RE workflow (binwalk, FACT, EMBA) for embedded device analysis, and obfuscated code analysis (LLVM obfuscation, OLLVM, Tigress). This domain covers modern program-analysis techniques that scale beyond manual reverse engineering, with industry-standard tooling workflows. Distinct from foundational binary-reverse (which covers basic radare2 / Ghidra introduction) — this skill focuses on advanced program analysis, automated RE pipelines, and firmware / obfuscation workflows.
Key Terms
Symbolic execution — Program analysis technique that explores all paths via symbolic variables
SMT solver — Satisfiability Modulo Theories solver (Z3) used by symbolic execution
Binary diffing (BinDiff / Diaphora / Kam1n0) for variant analysis
Firmware RE workflow (binwalk / FACT / EMBA)
Obfuscated code analysis (LLVM / OLLVM / Tigress)
Decompiler confusion + deobfuscation
APT-grade analysis (Equation Group, Pegasus)
Out of scope: foundational RE (see binary-reverse), malware analysis workflow (see malware-analysis-advanced), exploit development (see exploit-development).
Use Cases
Symbolic execution for key validation: Recover algorithm via SMT solving
Firmware RE for routers / IoT: Extract filesystem + analyze embedded services
OLLVM deobfuscation: Defeat Control Flow Flattening + Bogus Control Flow
Decompiler-resistant code analysis: Manual disassembly when decompiler fails
Variant analysis: Identify family of malware / binaries via diffing
SMT-assisted key recovery: Recover cryptographic keys via Z3
Automated RE pipeline: Build CI/CD for binary analysis
Equation Group / Pegasus analysis: APT-grade obfuscation research
Embedded device security: Audit router / IoT / OT firmware
Core Tools
Tool
Purpose
angr
Python symbolic execution framework
KLEE
LLVM-based symbolic execution
manticore
Symbolic execution (Trail of Bits)
Ghidra
NSA open-source RE tool
IDA Pro
Industry-standard disassembler + decompiler
Binary Ninja
Modern disassembler with rich API
radare2
Open-source disassembler
binwalk
Firmware analysis tool
BinDiff
Binary diffing (Google/Zynamics)
Diaphora
Free BinDiff alternative (IDA plugin)
Kam1n0
Binary similarity (assembly)
FACT
Firmware Analysis Compare Tool
EMBA
Embedded firmware analyzer
ollvm-tools
OLLVM deobfuscation tools
deflat
Control Flow Flattening deobfuscation
snowman-decompiler
Open-source decompiler
retdec
Avast open-source decompiler
imhex
Modern hex editor
pe-tree
Visual PE analysis
ida-deobfuscator
IDA plugin for deobfuscation
Methodology
Phase 1 — Static triage
file binary
sha256sum binary
strings binary | head -20
# Architecture
file binary
# Imported functions
nm -D binary 2>/dev/null | head
readelf -d binary 2>/dev/null | head# Section entropy (packed indicator)
python3 -c "
import sys
with open('binary', 'rb') as f:
data = f.read()
import math
entropy = -sum((data.count(b)/len(data)) * math.log2(data.count(b)/len(data)) for b in set(data))
print(f'Entropy: {entropy:.2f}')
"
Phase 2 — Binary diffing
# BinDiff (Google)
bindiff --binary1=v1.exe --binary2=v2.exe --output_dir=diffs/
# Diaphora (IDA plugin)# 1. Open v1.exe in IDA → Export with Diaphora# 2. Open v2.exe in IDA → Diff with Diaphora# Patch diff (CVE analysis)# 1. Get pre-patch binary# 2. Get post-patch binary# 3. BinDiff / Diaphora to identify changed functions# 4. Analyze changed function for CVE
file binary
sha256sum binary
strings binary | head
python3 -c "
import pefile
pe = pefile.PE('binary.exe')
for s in pe.sections:
print(s.Name.decode().rstrip(chr(0)), s.get_entropy())
"
Step 2 — Symbolic execution with angr
import angr
proj = angr.Project('./crackme', auto_load_libs=False)
state = proj.factory.entry_state()
# Find / avoid
sm = proj.factory.simulation_manager(state)
sm.explore(find=lambda s: b'Good boy'in s.posix.dumps(1),
avoid=lambda s: b'Bad boy'in s.posix.dumps(1))
if sm.found:
found = sm.found[0]
print(f"Password: {found.posix.dumps(0)}")
Step 3 — BinDiff for variant analysis
bindiff --binary1=original --binary2=patched --output_dir=diffs
# Analyze resultscd diffs
ls# original_patched.Diff → open in BinDiff UI
# Identify dispatcher function# Look for big switch statement on state variable# Use deflat.py (https://github.com/cd70s062f/deflat)
python3 deflat.py --binary flattened.exe --dispatcher 0x401000
Step 6 — SMT key recovery
import z3
# Encode key check
s = z3.Solver()
# Input: 16-byte key
key = [z3.BitVec(f'key_{i}', 8) for i inrange(16)]
# Constraintsfor i inrange(16):
s.add(key[i] >= 0x20)
s.add(key[i] <= 0x7e)
# Key check (derived from disassembly)
s.add(key[0] + key[1] == 0x90)
s.add(key[2] * key[3] == 0x41A8)
# ...if s.check() == z3.sat:
m = s.model()
print(bytes(m[k].as_long() for k in key))
Step 7 — Ghidra decompile
analyzeHeadless /tmp ghidra_proj -import binary
# Then open GUI
ghidraRun
