ctf-reverse

name	ctf-reverse
description	Reverse engineering: ELF/PE/Mach-O, WASM, .NET, APK (Flutter/Dart), Python bytecode, Go/Rust/Swift/Kotlin, custom VMs, anti-debug/anti-VM, VMProtect/Themida, eBPF, Ghidra/IDA/radare2/Frida/angr/Qiling. Dispatch on file magic + loader signature.

CTF Reverse Engineering

Quick reference for RE challenges. For detailed techniques, see supporting files.

Additional Resources

• tools.md — GDB, Ghidra, radare2, IDA, Binary Ninja, Unicorn, WASM, pyc, packed
• tools-dynamic.md — Frida, angr, lldb, x64dbg, Qiling, Triton, Pin instruction-counting
• tools-advanced.md — VMProtect/Themida, BinDiff, D-810/GOOMBA, TTF GSUB, AVX2 Z3 lift
• tools-advanced-2.md — 2025-26: GB-scale PE Unicorn+angr hybrid (VirtualProtect-gated unpackers)
• anti-analysis.md — Linux/Windows anti-debug, anti-VM, anti-DBI, MBA, self-hashing
• patterns.md — custom VMs, nanomites, LLVM obfuscation, S-box, SECCOMP/BPF, multi-thread
• patterns-ctf.md — comp patterns part 1: hidden opcodes, LD_PRELOAD, GBA MITM, maze kmod
• patterns-ctf-2.md — part 2: multi-layer brute, CVP integer, decision-tree, perf oracle, VM misident
• patterns-ctf-3.md — 2025-26: genetic algorithm / hill-climb over opaque additive scoring
• languages.md — Python bytecode, pyarmor, UEFI, esolangs, HarmonyOS, Godot, Electron
• languages-compiled.md — Go (GoReSym), Rust, Swift, Kotlin/JVM, C++ vtables, .pyc forgery
• platforms.md — Mach-O, iOS jailbreak, embedded firmware, kernel drivers, game engines, CAN

---

Pattern Recognition Index

Dispatch on observable binary features, not challenge titles.

Signal (from file, readelf, strings, nm)	Technique → file
ELF with __libc_start_main, small main, direct syscalls	Basic RE patterns → patterns.md
ELF with large unrecognised opcode-dispatch loop (switch on byte → handler)	Custom VM reversing → patterns.md
readelf -l shows RWX segment + self-writes to .text	Self-modifying / multi-layer decryption → patterns-ctf-2.md
Binary that modifies its round constants and re-encrypts output	Binary-as-keystream-oracle (patch I/O boundary) → patterns-ctf-2.md
ptrace(PTRACE_TRACEME) / /proc/self/status TracerPid / rdtsc timing	Anti-debug detection → anti-analysis.md
__Py_* or PyMarshal strings	Python bytecode / pyc reversing → languages.md
runtime. prefix in strings, go.buildinfo	Go reversing (GoReSym) → languages-compiled.md
Rust demangling (_ZN/_RN), core::panicking	Rust reversing → languages-compiled.md
Mach-O header FEEDFACE/FEEDFACF	macOS/iOS RE → platforms.md
.wasm magic (00 61 73 6D)	WASM → languages.md, ctf-misc/games-and-vms.md
.apk/classes.dex, libflutter.so, kernel.dill	APK / Flutter reversing → languages.md
Unicorn/QEMU used as a sandbox with host-side memory read helpers	Host/guest hook divergence → patterns-ctf-2.md (and ctf-pwn/advanced-exploits-2.md)
.rodata blob + XOR loop with known constants / stored expected bytes	Stack-string deobfuscation → patterns-ctf-2.md
SHA-NI instructions, per-layer key read from stdin	Multi-layer brute-force JIT → patterns-ctf-2.md
Per-char early-exit compare loop + local execution allowed	perf_event_open instruction-count oracle → patterns-ctf-2.md
Custom VM whose handlers are pop/push but docs claim "register-based" + banned bytes	Arch misidentification + banned-byte synthesis → patterns-ctf-2.md
.pyc with loader that checks only first 16 bytes	PEP-552 magic-header forgery → languages-compiled.md
Go binary with runtime.itab symbols intact but stripped strings	GoReSym/typelinks restore → languages-compiled.md
bpftool prog list shows non-standard eBPF prog	eBPF FSM syscall-sequence decomp → languages-compiled.md (+ ctf-pwn/sandbox-escape.md)
TTF/OTF with abnormally dense GSUB; glyphs named hex_*/one/zero	GSUB ligature stego DAG reverse → tools-advanced.md
AVX2 vpaddb/vpshufb in tight loop over input	Lane-wise Z3 lifting → tools-advanced.md
PE ≥ 500 MB, multiple VirtualProtect(...,RWX) + inline decrypt + call/jmp rax after each	Unicorn layer-graph + per-layer angr solve → tools-advanced-2.md
Flat chain of hundreds of if (input[i] op const) score += kN; win if score >= THR	Separability probe → hill-climb → GA → patterns-ctf-3.md

Recognize the artefact or opcode pattern. The title is noise.

---

For inline code/cheatsheet quick references (grep patterns, one-liners, common payloads), see quickref.md. The Pattern Recognition Index above is the dispatch table — always consult it first; load quickref.md only if you need a concrete snippet after dispatch.

---

CTF Reverse - Anti-Analysis Techniques & Bypasses

Comprehensive reference for anti-debugging, anti-VM, anti-DBI, and integrity-check techniques encountered in CTF challenges, with practical bypasses.

Table of Contents

• Linux Anti-Debug (Advanced)
  • ptrace-Based
  • /proc Filesystem Checks
  • Timing-Based Detection
  • Signal-Based Anti-Debug
  • Syscall-Level Evasion
• Windows Anti-Debug (Advanced)
  • PEB-Based Checks
  • NtQueryInformationProcess
  • Heap Flags
  • TLS Callbacks
  • Hardware Breakpoint Detection
  • Software Breakpoint Detection (INT3 Scanning)
  • Exception-Based Anti-Debug
  • NtSetInformationThread (Thread Hiding)
• Anti-VM / Anti-Sandbox
  • CPUID Hypervisor Bit
  • MAC Address / Hardware Fingerprinting
  • Timing-Based VM Detection
  • File / Registry Artifacts
  • Resource Checks (CPU Count, RAM, Disk)
• Anti-DBI (Dynamic Binary Instrumentation)
  • Frida Detection
  • Pin/DynamoRIO Detection
• Code Integrity / Self-Hashing
• Anti-Disassembly Techniques
  • Opaque Predicates
  • Junk Bytes / Overlapping Instructions
  • Jump-in-the-Middle
  • Function Chunking / Scattered Code
  • Control Flow Flattening (Advanced)
  • Mixed Boolean-Arithmetic (MBA) Identification & Simplification
• Comprehensive Bypass Strategies
  • Universal Bypass Checklist
  • Layered Anti-Debug (Real-World Pattern)
  • Quick Reference: Check to Bypass

---

Linux Anti-Debug (Advanced)

ptrace-Based

Self-ptrace (most common):

if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) exit(1); // Already traced = debugger attached

Bypasses:

# 1. LD_PRELOAD (see patterns.md for full hook)
LD_PRELOAD=./hook.so ./binary

# 2. Patch with pwntools
python3 -c "
from pwn import *
elf = ELF('./binary', checksec=False)
elf.asm(elf.symbols.ptrace, 'xor eax, eax; ret')
elf.save('patched')
"

# 3. GDB: catch the syscall
gdb ./binary
(gdb) catch syscall ptrace
(gdb) run
# When it stops at ptrace:
(gdb) set $rax = 0
(gdb) continue

# 4. Kernel config (requires root)
echo 0 > /proc/sys/kernel/yama/ptrace_scope

Double-ptrace pattern:

// Fork child to ptrace parent — blocks all other debuggers
pid_t child = fork();
if (child == 0) {
    ptrace(PTRACE_ATTACH, getppid(), 0, 0);
    // Child sits in waitpid loop, keeping parent traced
} else {
    // Parent continues with real logic
}

Bypass: Kill the watchdog child process, then attach debugger.

/proc Filesystem Checks

// TracerPid check
FILE *f = fopen("/proc/self/status", "r");
// Looks for "TracerPid:\t0" — non-zero means debugger

// /proc/self/exe link check (some debuggers change this)
readlink("/proc/self/exe", buf, sizeof(buf));

// /proc/self/maps — check for debugger libraries
grep("frida", "/proc/self/maps");

Bypasses:

# 1. LD_PRELOAD fopen/fread to fake /proc contents
# 2. Mount namespace isolation
unshare -m bash -c 'mount --bind /dev/null /proc/self/status && ./binary'

# 3. GDB: set breakpoint at fopen, change filename argument
(gdb) b fopen
(gdb) run
(gdb) set {char[20]} $rdi = "/dev/null"
(gdb) continue

Timing-Based Detection

// rdtsc (CPU timestamp counter)
uint64_t start = __rdtsc();
// ... code ...
uint64_t delta = __rdtsc() - start;
if (delta > THRESHOLD) exit(1);  // too slow = debugger

// clock_gettime
struct timespec ts1, ts2;
clock_gettime(CLOCK_MONOTONIC, &ts1);
// ... code ...
clock_gettime(CLOCK_MONOTONIC, &ts2);

// gettimeofday
struct timeval tv1, tv2;
gettimeofday(&tv1, NULL);

Bypasses:

# 1. Frida hook (see tools-dynamic.md for clock_gettime hook)

# 2. GDB: skip rdtsc by patching with constant
(gdb) set {unsigned char[2]} 0x401234 = {0x90, 0x90}  # NOP the rdtsc

# 3. Pin tool to fix TSC reads
# 4. faketime library
LD_PRELOAD=/usr/lib/faketime/libfaketime.so.1 FAKETIME="2024-01-01" ./binary

Signal-Based Anti-Debug

// SIGTRAP handler — INT3 under debugger is caught by debugger, not handler
signal(SIGTRAP, handler);
__asm__("int3");
// If handler runs: no debugger. If debugger catches: debugged.

// SIGALRM timeout — kill self if analysis takes too long
signal(SIGALRM, kill_handler);
alarm(5);

// SIGSEGV handler that does real work (see patterns.md for MBA pattern)
signal(SIGSEGV, real_logic_handler);
*(int*)0 = 0;  // deliberate crash → handler runs real code

Bypasses:

# GDB: pass signals to program instead of handling them
(gdb) handle SIGTRAP nostop pass
(gdb) handle SIGALRM ignore
(gdb) handle SIGSEGV nostop pass

# For alarm-based: patch alarm() to return immediately

Syscall-Level Evasion

// Direct syscall instead of libc — bypasses LD_PRELOAD hooks
long ret;
asm volatile("syscall" : "=a"(ret) : "a"(101), "D"(0), "S"(0), "d"(0), "r"(0));
// Syscall 101 = ptrace on x86_64

Bypass: Must patch the binary itself or use ptrace to intercept at syscall level.

# GDB: catch syscall
(gdb) catch syscall 101
(gdb) commands
> set $rax = 0
> continue
> end

---

Windows Anti-Debug (Advanced)

PEB-Based Checks

// BeingDebugged flag (offset 0x2 in PEB)
bool debugged = NtCurrentPeb()->BeingDebugged;

// NtGlobalFlag (offset 0x68/0xBC in PEB)
// When debugger: FLG_HEAP_ENABLE_TAIL_CHECK | FLG_HEAP_ENABLE_FREE_CHECK | FLG_HEAP_VALIDATE_PARAMETERS = 0x70
DWORD flags = *(DWORD*)((BYTE*)NtCurrentPeb() + 0xBC); // 64-bit offset
if (flags & 0x70) exit(1);

Bypass (x64dbg):

# ScyllaHide plugin auto-patches PEB fields
# Manual: dump PEB, zero BeingDebugged and NtGlobalFlag

NtQueryInformationProcess

// ProcessDebugPort (0x7)
DWORD_PTR debugPort = 0;
NtQueryInformationProcess(GetCurrentProcess(), 7, &debugPort, sizeof(debugPort), NULL);
if (debugPort != 0) exit(1);

// ProcessDebugObjectHandle (0x1E)
HANDLE debugObj = NULL;
NTSTATUS status = NtQueryInformationProcess(GetCurrentProcess(), 0x1E, &debugObj, sizeof(debugObj), NULL);
if (status == 0) exit(1); // STATUS_SUCCESS means debugger present

// ProcessDebugFlags (0x1F) — returns inverse: 0 = debugger present
DWORD noDebug = 0;
NtQueryInformationProcess(GetCurrentProcess(), 0x1F, &noDebug, sizeof(noDebug), NULL);
if (noDebug == 0) exit(1);

Bypass: Hook NtQueryInformationProcess to return fake values, or use ScyllaHide.

Heap Flags

// Process heap has debug flags when debugger attached
PHEAP heap = (PHEAP)GetProcessHeap();
// Flags at offset 0x70 (64-bit): should be HEAP_GROWABLE (0x2)
// ForceFlags at offset 0x74: should be 0
if (heap->Flags != 0x2 || heap->ForceFlags != 0) exit(1);

TLS Callbacks

Key technique: TLS (Thread Local Storage) callbacks execute BEFORE main() / entry point.

// Registered in PE header's TLS directory
void NTAPI TlsCallback(PVOID DllHandle, DWORD Reason, PVOID Reserved) {
    if (Reason == DLL_PROCESS_ATTACH) {
        if (IsDebuggerPresent()) {
            ExitProcess(1);  // Kills process before main runs
        }
    }
}

#pragma comment(linker, "/INCLUDE:_tls_used")
#pragma data_seg(".CRT$XLB")
PIMAGE_TLS_CALLBACK callbacks[] = { TlsCallback, NULL };

Detection in IDA/Ghidra: Check PE TLS Directory → AddressOfCallBacks. Functions listed there run before EP.

Bypass: Set breakpoint on TLS callback in x64dbg (Options → Events → TLS Callbacks), or patch the TLS directory entry.

Hardware Breakpoint Detection

// Read debug registers via GetThreadContext
CONTEXT ctx;
ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;
GetThreadContext(GetCurrentThread(), &ctx);
if (ctx.Dr0 || ctx.Dr1 || ctx.Dr2 || ctx.Dr3) exit(1);

// Also via exception handler: deliberate exception, check DR regs in handler

Bypass:

# x64dbg: use software breakpoints instead, or hook GetThreadContext
# Frida: hook GetThreadContext to zero DR registers

Software Breakpoint Detection (INT3 Scanning)

// CRC / hash check over code section
unsigned char *code = (unsigned char*)function_addr;
uint32_t checksum = 0;
for (int i = 0; i < code_size; i++) {
    checksum += code[i];
    if (code[i] == 0xCC) exit(1);  // INT3 = software breakpoint
}
if (checksum != EXPECTED_CHECKSUM) exit(1);

Bypass: Use hardware breakpoints (DR0-DR3) instead of software breakpoints. Or hook the scanning function.

Exception-Based Anti-Debug

// UnhandledExceptionFilter — under debugger, filter is NOT called
SetUnhandledExceptionFilter(handler);
RaiseException(EXCEPTION_ACCESS_VIOLATION, 0, 0, NULL);
// If handler runs: no debugger
// If debugger catches: debugger present

// INT 2D — debugger single-step anomaly
__asm { int 2dh }  // Debugger silently consumes the exception
// If execution continues: debugger present

NtSetInformationThread (Thread Hiding)

// Hide thread from debugger — stops all debug events
typedef NTSTATUS(NTAPI *pNtSIT)(HANDLE, ULONG, PVOID, ULONG);
pNtSIT NtSIT = (pNtSIT)GetProcAddress(GetModuleHandle("ntdll"), "NtSetInformationThread");
NtSIT(GetCurrentThread(), 0x11 /*ThreadHideFromDebugger*/, NULL, 0);
// After this, debugger won't see breakpoints or exceptions from this thread

Bypass: Hook NtSetInformationThread to ignore class 0x11, or patch the call.

---

Anti-VM / Anti-Sandbox

CPUID Hypervisor Bit

int regs[4];
__cpuid(regs, 1);
if (regs[2] & (1 << 31)) {  // ECX bit 31 = hypervisor present
    exit(1);
}

// Hypervisor brand string
__cpuid(regs, 0x40000000);
char brand[13] = {0};
memcpy(brand, &regs[1], 12);
// "VMwareVMware", "Microsoft Hv", "KVMKVMKVM", "XenVMMXenVMM"

Bypass: Patch cpuid results or use LD_PRELOAD to hook wrapper functions.

MAC Address / Hardware Fingerprinting

Known VM MAC prefixes:
  VMware:     00:0C:29, 00:50:56
  VirtualBox: 08:00:27
  Hyper-V:    00:15:5D
  Parallels:  00:1C:42
  QEMU:       52:54:00

Timing-Based VM Detection

// VM exits on privileged instructions are measurably slower
uint64_t start = __rdtsc();
__cpuid(regs, 0);  // Forces VM exit
uint64_t delta = __rdtsc() - start;
if (delta > 500) { /* likely VM */ }

File / Registry Artifacts

Files: C:\Windows\System32\drivers\vm*.sys, vbox*.dll, VBoxService.exe
Registry: HKLM\SOFTWARE\VMware, Inc.\VMware Tools
Services: VMTools, VBoxService
Processes: vmtoolsd.exe, VBoxTray.exe, qemu-ga.exe
Linux: /sys/class/dmi/id/product_name contains "VirtualBox"|"VMware"
       dmesg | grep -i "hypervisor detected"

Resource Checks (CPU Count, RAM, Disk)

// Sandboxes typically have minimal resources
SYSTEM_INFO si;
GetSystemInfo(&si);
if (si.dwNumberOfProcessors < 2) exit(1);

MEMORYSTATUSEX ms;
ms.dwLength = sizeof(ms);
GlobalMemoryStatusEx(&ms);
if (ms.ullTotalPhys < 2ULL * 1024 * 1024 * 1024) exit(1); // < 2GB RAM

// Disk size check (< 60GB = sandbox)
GetDiskFreeSpaceEx("C:\\", NULL, &total, NULL);

Bypass: Use a VM configured with adequate resources (4+ CPUs, 8GB+ RAM, 100GB+ disk).

---

Anti-DBI (Dynamic Binary Instrumentation)

Frida Detection

// 1. Check /proc/self/maps for frida-agent
FILE *f = fopen("/proc/self/maps", "r");
while (fgets(line, sizeof(line), f)) {
    if (strstr(line, "frida") || strstr(line, "gadget")) exit(1);
}

// 2. Check for Frida's default port (27042)
int sock = socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in addr = {.sin_family=AF_INET, .sin_port=htons(27042), .sin_addr.s_addr=inet_addr("127.0.0.1")};
if (connect(sock, (struct sockaddr*)&addr, sizeof(addr)) == 0) exit(1);

// 3. Check for inline hooks (function prologue modification)
// Compare first bytes of libc functions against expected values
unsigned char *strcmp_bytes = (unsigned char *)strcmp;
if (strcmp_bytes[0] == 0xE9 || strcmp_bytes[0] == 0xFF) exit(1); // JMP = hooked

// 4. Thread name check
// Frida creates threads with names like "gmain", "gdbus", "frida-*"
DIR *dir = opendir("/proc/self/task");
while ((entry = readdir(dir))) {
    char comm_path[256];
    snprintf(comm_path, sizeof(comm_path), "/proc/self/task/%s/comm", entry->d_name);
    // Read comm and check for "gmain", "gdbus"
}

// 5. Named pipe detection (Windows)
// Frida creates \\.\pipe\frida-* named pipes

Frida bypass of Frida detection:

// Hook the detection functions themselves
Interceptor.attach(Module.findExportByName(null, "strstr"), {
    onEnter(args) {
        this.haystack = Memory.readUtf8String(args[0]);
        this.needle = Memory.readUtf8String(args[1]);
    },
    onLeave(retval) {
        if (this.needle && (this.needle.includes("frida") || this.needle.includes("gadget"))) {
            retval.replace(ptr(0)); // Not found
        }
    }
});

// Early Frida load (before anti-DBI runs)
// Use frida-gadget as early-init shared library

Pin/DynamoRIO Detection

// Check for instrumentation libraries in /proc/self/maps
// Pin: "pin-", "pinbin", "pinatrace"
// DynamoRIO: "dynamorio", "drcov", "drrun"

// Instruction count timing — DBI adds overhead
// Execute known instruction sequence, compare execution time

---

Code Integrity / Self-Hashing

// CRC32 over .text section
uint32_t crc = compute_crc32(text_start, text_size);
if (crc != EXPECTED_CRC) exit(1);  // Code was modified (breakpoints, patches)

// MD5/SHA256 of function bodies
unsigned char hash[32];
SHA256(function_addr, function_size, hash);
if (memcmp(hash, expected_hash, 32) != 0) exit(1);

Bypasses:

1. Hardware breakpoints (don't modify code, DR0-DR3)
2. Patch the comparison to always succeed
3. Hook the hash function to return expected value
4. Emulate instead of debug (Unicorn/Qiling — no code modification)
5. Snapshot + restore: dump memory before and after, diff to find checks

Self-checksumming in loops:

// Continuous integrity check in separate thread
void *watchdog(void *arg) {
    while (1) {
        if (compute_crc32(text_start, text_end - text_start) != saved_crc) {
            memset(flag_buffer, 0, flag_len);  // Destroy flag
            exit(1);
        }
        usleep(100000);
    }
}

Bypass: Kill the watchdog thread or patch its sleep to infinite.

---

Anti-Disassembly Techniques

Opaque Predicates

; Condition that always evaluates the same way but looks data-dependent
mov eax, [some_memory]
imul eax, eax          ; x^2
and eax, 1             ; x^2 mod 2 is always 0 for any x
jnz fake_branch        ; Never taken, but disassembler doesn't know
; real code here

Identification: Z3/SMT can prove branch is always/never taken.

Junk Bytes / Overlapping Instructions

jmp real_code
db 0xE8           ; Looks like start of CALL to linear disassembler
real_code:
mov eax, 1        ; Real code — disassembler may misalign here

Fix: Switch to graph-mode disassembly (Ghidra/IDA handle this well). Manual: undefine and re-analyze from correct offset.

Jump-in-the-Middle

; Jumps into the middle of a multi-byte instruction
eb 01          ; jmp +1 (skip next byte)
e8             ; fake CALL opcode — disassembler tries to decode as call
90             ; real: NOP (landed here from jmp)

Function Chunking / Scattered Code

Functions split into non-contiguous chunks connected by unconditional jumps. Defeats linear function boundary detection.

Tool: IDA's "Append function tail" or Ghidra's "Create function" at each chunk.

Control Flow Flattening (Advanced)

Beyond basic switch-case (see patterns.md): modern OLLVM variants use:

• Bogus control flow: Fake branches with opaque predicates
• Instruction substitution: a + b → a - (-b), a ^ b → (a | b) & ~(a & b)
• String encryption: Strings decrypted at runtime, cleared after use

Deobfuscation tools:

• D-810 (IDA plugin): Pattern-based deobfuscation, MBA simplification
• GOOMBA (Ghidra): Automated deobfuscation for OLLVM
• Miasm: Symbolic execution for deobfuscation
• Arybo / SiMBA: MBA expression simplification

# D-810: install in IDA plugins directory, Edit → Plugins → D-810
# Simplifies MBA expressions: (a | b) & ~(a & b) → a ^ b
# Removes opaque predicates via pattern matching

Mixed Boolean-Arithmetic (MBA) Identification & Simplification

# Common MBA patterns and their simplified forms:
# (x & y) + (x | y) == x + y
# (x ^ y) + 2*(x & y) == x + y
# (x | y) - (x & ~y) == y
# ~(~x & ~y) == x | y (De Morgan's)
# (x | y) & ~(x & y) == x ^ y

# SiMBA tool for automated simplification:
# pip install simba-simplifier
from simba import simplify_mba
expr = "(a | b) + (a & b) - (~a & b)"
print(simplify_mba(expr))  # → a

---

Comprehensive Bypass Strategies

Universal Bypass Checklist

1. Identify all anti-analysis checks — search for: ptrace, IsDebuggerPresent, rdtsc, cpuid, NtQuery, GetTickCount, CheckRemoteDebuggerPresent, /proc/self, SIGTRAP, alarm
2. Static patching — NOP/patch checks with pwntools or Ghidra before running
3. LD_PRELOAD (Linux) — hook libc functions returning fake values
4. ScyllaHide (Windows x64dbg) — patches PEB, hooks NT functions automatically
5. Emulation (Unicorn/Qiling) — no debugger artifacts to detect
6. Kernel-level bypass — modify /proc/sys/kernel/yama/ptrace_scope, use prctl

Layered Anti-Debug (Real-World Pattern)

Many CTF challenges stack multiple checks:

1. TLS callback → IsDebuggerPresent (before main)
2. main() → ptrace(TRACEME)
3. Watchdog thread → timing check + /proc scan
4. Code section → self-CRC32 integrity
5. Signal handler → real logic in SIGSEGV handler

Approach: Identify ALL checks before patching. Patch or hook each one systematically. Run under emulator if too many to patch individually.

Quick Reference: Check to Bypass

Anti-Debug Check	Platform	Bypass
ptrace(TRACEME)	Linux	LD_PRELOAD, patch to ret 0, catch syscall
IsDebuggerPresent	Windows	ScyllaHide, Frida hook, PEB patch
NtQueryInformationProcess	Windows	ScyllaHide, hook ntdll
rdtsc timing	Both	NOP rdtsc, Frida time hook, Pin
/proc/self/status	Linux	Mount namespace, hook fopen
alarm(N)	Linux	handle SIGALRM ignore in GDB
SIGTRAP handler	Linux	handle SIGTRAP nostop pass
TLS callback	Windows	Break on TLS in x64dbg, patch
DR register scan	Windows	Use software BPs, hook GetThreadContext
INT3 scan / CRC	Both	Hardware BPs, patch CRC comparison
Frida detection	Both	Early-load gadget, hook strstr
CPUID hypervisor	Both	Patch CPUID result, bare metal
Thread hiding	Windows	Hook NtSetInformationThread

---

CTF Reverse - Compiled Language Reversing (Go, Rust)

Table of Contents

• Go Binary Reversing
  • Recognition
  • Symbol Recovery
  • Go Memory Layout
  • Goroutine and Concurrency Analysis
  • Common Go Patterns in Decompilation
  • Go Binary Reversing Workflow
• Rust Binary Reversing
  • Recognition
  • Symbol Demangling
  • Common Rust Patterns in Decompilation
  • Rust-Specific Analysis Tools
• Swift Binary Reversing
• Kotlin / JVM Binary Reversing
  • JVM Bytecode (Android/Server)
  • Kotlin/Native
• C++ Binary Reversing (Quick Reference)
  • vtable Reconstruction
  • RTTI (Run-Time Type Information)
  • Standard Library Patterns

---

Go Binary Reversing

Go binaries are increasingly common in CTF challenges due to Go's popularity for CLI tools, network services, and malware.

Recognition

# Detect Go binary
file binary | grep -i "go"
strings binary | grep "go.buildid"
strings binary | grep "runtime.gopanic"

# Go version embedded in binary
strings binary | grep "^go1\."

Key indicators:

• Very large static binary (even "hello world" is ~2MB)
• Embedded go.buildid string
• runtime.* symbols (even in stripped binaries, some remain)
• main.main as entry point (not main)
• Strings like GOROOT, GOPATH, /usr/local/go/src/

Symbol Recovery

Go embeds rich type and function information even in stripped binaries:

# GoReSym - recovers function names, types, interfaces from Go binaries
# https://github.com/mandiant/GoReSym
./GoReSym -d binary > symbols.json

# Parse output
python3 -c "
import json
with open('symbols.json') as f:
    data = json.load(f)
for fn in data.get('UserFunctions', []):
    print(f\"{fn['Start']:#x}  {fn['FullName']}\")
"

Ghidra with golang-loader:

# Install: Ghidra → Window → Script Manager → search "golang"
# Or use: https://github.com/getCUJO/ThreatFox/tree/main/ghidra-golang
# Recovers function names, string references, interface tables

redress (Go binary analysis):

# https://github.com/goretk/redress
redress -src binary         # Reconstruct source tree
redress -pkg binary         # List packages
redress -type binary        # List types and methods
redress -interface binary   # List interfaces

Go Memory Layout

Understanding Go's data structures in decompilation:

# String: {pointer, length} (16 bytes on 64-bit)
# NOT null-terminated! Length field is critical.
struct GoString {
    char *ptr;    // pointer to UTF-8 data
    int64 len;    // byte length
};

# Slice: {pointer, length, capacity} (24 bytes on 64-bit)
struct GoSlice {
    void *ptr;    // pointer to backing array
    int64 len;    // current length
    int64 cap;    // allocated capacity
};

# Interface: {type_descriptor, data_pointer} (16 bytes)
struct GoInterface {
    void *type;   // points to type metadata (itab for non-empty interface)
    void *data;   // points to actual value
};

# Map: pointer to runtime.hmap struct
# Channel: pointer to runtime.hchan struct

In Ghidra/IDA: When you see a function taking (ptr, int64) — it's likely a Go string. Three-field (ptr, int64, int64) is a slice.

Goroutine and Concurrency Analysis

# Identify goroutine spawns in disassembly
strings binary | grep "runtime.newproc"
# newproc1 is the internal goroutine creation function

# In GDB with Go support:
gdb ./binary
(gdb) source /usr/local/go/src/runtime/runtime-gdb.py
(gdb) info goroutines          # List all goroutines
(gdb) goroutine 1 bt          # Backtrace for goroutine 1

Channel operations in disassembly:

• runtime.chansend1 → ch <- value
• runtime.chanrecv1 → value = <-ch
• runtime.selectgo → select { case ... }
• runtime.closechan → close(ch)

Common Go Patterns in Decompilation

Defer mechanism:

• runtime.deferproc → registers deferred function
• runtime.deferreturn → executes deferred functions at function exit
• Deferred calls execute in LIFO order — relevant for cleanup/crypto key wiping

Error handling (the if err != nil pattern):

# In disassembly, this appears as:
# call some_function        → returns (result, error) as two values
# test rax, rax             → check if error (second return value) is nil
# jne error_handler

String concatenation:

• runtime.concatstrings → s1 + s2 + s3
• fmt.Sprintf → formatted string building
• Look for format strings in .rodata: "%s%d", "%x"

Common stdlib patterns in CTF:

// Crypto operations → look for these in strings/imports:
// "crypto/aes", "crypto/cipher", "crypto/sha256", "encoding/hex", "encoding/base64"

// Network operations:
// "net/http", "net.Dial", "bufio.NewReader"

// File operations:
// "os.Open", "io.ReadAll", "os.ReadFile"

Go Binary Reversing Workflow

1. file binary                          # Confirm Go, get arch
2. GoReSym -d binary > syms.json       # Recover symbols
3. strings binary | grep -i flag        # Quick win check
4. Load in Ghidra with golang-loader    # Apply recovered symbols
5. Find main.main                       # Entry point
6. Identify string comparisons          # GoString {ptr, len} pairs
7. Trace crypto operations              # crypto/* package usage
8. Check for embedded resources         # embed.FS in Go 1.16+

Go embed.FS (Go 1.16+): Binaries can embed files at compile time:

# Look for embedded file data
strings binary | grep "embed"
# Embedded files appear as raw data in the binary
# Search for known file signatures (PK for zip, PNG header, etc.)

Key insight: Go's runtime embeds extensive metadata even in stripped binaries. Use GoReSym before any manual analysis — it often recovers 90%+ of function names, making decompilation dramatically easier. Go strings are {ptr, len} tuples, not null-terminated — Ghidra's default string analysis will miss them without the golang-loader plugin.

Detection: Large static binary (2MB+ for simple programs), go.buildid, runtime.gopanic, source paths like /home/user/go/src/.

---

Rust Binary Reversing

Rust binaries are common in modern CTFs, especially for crypto, systems, and security tooling challenges.

Recognition

# Detect Rust binary
strings binary | grep -c "rust"
strings binary | grep "rustc"             # Compiler version
strings binary | grep "/rustc/"           # Source paths
strings binary | grep "core::panicking"   # Panic infrastructure

Key indicators:

• core::panicking::panic in strings
• Mangled symbols starting with _ZN (Itanium ABI) — e.g., _ZN4main4main17h...
• .rustc section in ELF
• References to /rustc/<commit_hash>/library/
• Large binary size (Rust statically links by default)

Symbol Demangling

# Rust uses Itanium ABI mangling (same as C++)
# rustfilt demangles Rust-specific symbols
cargo install rustfilt
nm binary | rustfilt | grep "main"

# Or use c++filt (works for most Rust symbols)
nm binary | c++filt | grep "main"

# In Ghidra: Window → Script Manager → search "Demangler"
# Enable "DemangleAllScript" for automatic demangling

Common Rust Patterns in Decompilation

Option/Result enum:

# Option<T> in memory: {discriminant (0=None, 1=Some), value}
# Result<T, E>: {discriminant (0=Ok, 1=Err), union{ok_val, err_val}}

# In disassembly:
# cmp byte [rbp-0x10], 0    → check if None/Err
# je handle_none_case

Vec (same as Go slice):

struct RustVec {
    void *ptr;      // heap pointer
    uint64 cap;     // capacity
    uint64 len;     // length
};

String / &str:

# String (owned): {ptr, capacity, length} — 24 bytes, heap-allocated
# &str (borrowed): {ptr, length} — 16 bytes, can point anywhere

# In decompilation, look for:
# alloc::string::String::from    → String creation
# core::str::from_utf8           → byte slice to str

Iterator chains:

# .iter().map().filter().collect() compiles to loop fusion
# In disassembly: tight loop with inlined closures
# Look for: core::iter::adapters::map, filter, etc.

Panic unwinding:

# Panic strings reveal source locations and error messages
strings binary | grep "panicked at"
strings binary | grep "called .unwrap().. on"
# These often contain file paths, line numbers, and variable names

Rust-Specific Analysis Tools

# cargo-bloat: analyze binary size by function
cargo install cargo-bloat
cargo bloat --release -n 50

# Ghidra Rust helper scripts
# https://github.com/AmateursCTF/ghidra-rust (community scripts for Rust RE)

Key insight: Rust panic messages are goldmines — they contain source file paths, line numbers, and descriptive error strings even in release builds. Always strings binary | grep "panicked" first. Rust's monomorphization means generic functions get duplicated per type — expect many similar-looking functions.

Detection: core::panicking, .rustc section, /rustc/ paths, _ZN mangled symbols with Rust-style module paths.

---

Swift Binary Reversing

See platforms.md for full Swift reversing guide including demangling, runtime structures, and Ghidra integration. Key quick reference:

# Detect Swift binary
strings binary | grep "swift"
otool -l binary | grep "swift"

# Demangle Swift symbols
swift demangle 's14MyApp0A8ClassC10checkInput6resultSbSS_tF'
# → MyApp.MyAppClass.checkInput(result: String) -> Bool

# Key runtime functions: swift_allocObject, swift_release, swift_once
# String: small (≤15 bytes inline) or large (heap pointer + length)
# Protocol witness tables = dynamic dispatch (like vtables)

Detection: __swift5_* sections in Mach-O, swift_ runtime symbols, s prefix in mangled names.

---

Kotlin / JVM Binary Reversing

Kotlin compiles to JVM bytecode or native (via Kotlin/Native). Common in Android and server-side CTF.

JVM Bytecode (Android/Server)

# Detect Kotlin
strings classes.dex | grep "kotlin"
# Look for: kotlin.Metadata annotation, kotlin/jvm/internal/*

# Decompile
jadx classes.dex                     # Best for Kotlin bytecode
cfr classes.jar --kotlin             # CFR with Kotlin mode
fernflower classes.jar output/       # IntelliJ's decompiler

# Kotlin-specific patterns in decompiled output:
# - Companion objects: ClassName$Companion
# - Data classes: copy(), component1(), component2(), toString()
# - Coroutines: ContinuationImpl, invokeSuspend, state machine
# - Null checks: Intrinsics.checkNotNull() everywhere
# - When expression: compiled as tableswitch/lookupswitch
# - Sealed classes: instanceof checks in chain

Kotlin coroutines in disassembly:

# Coroutines compile to state machines:
# invokeSuspend(result) {
#     switch (this.label) {
#         case 0: this.label = 1; return suspendFunction();
#         case 1: processResult(result); return Unit;
#     }
# }
# Each suspend point becomes a state in the switch.
# Follow the state machine to understand async flow.

Kotlin/Native

# Kotlin/Native produces platform binaries (no JVM)
# Recognize by: konan, kotlin.native strings
strings binary | grep "konan"

# Much harder to reverse — no reflection metadata
# Uses LLVM backend, looks similar to C/C++ in disassembly
# Key functions: InitRuntime, DeinitRuntime, CreateStablePointer
# Memory management: automatic reference counting (not GC)

Detection: kotlin.Metadata annotations (JVM), konan strings (Native), kotlin/ package paths.

---

C++ Binary Reversing (Quick Reference)

While C++ RE is well-covered by general tools, these patterns are CTF-specific:

vtable Reconstruction

# Virtual function tables (vtables):
# First 8 bytes of object → pointer to vtable
# vtable entries: [typeinfo_ptr, destructor, method1, method2, ...]
# In Ghidra: Data → Create Pointer at vtable address

# Identify polymorphic dispatch:
# mov rax, [rdi]           # Load vtable from this pointer
# call [rax + 0x18]        # Call 4th virtual method (0x18/8 = 3rd after typeinfo+dtor)

RTTI (Run-Time Type Information)

# If not stripped, RTTI reveals class hierarchy
strings binary | grep -E "^[0-9]+[A-Z]"   # Mangled type names
c++filt _ZTI7MyClass                        # → typeinfo for MyClass

# In Ghidra: search for vtable references, follow typeinfo pointer
# typeinfo struct: {vtable_for_typeinfo, name_string, base_class_ptr}

Standard Library Patterns

std::string (libstdc++):
  SSO (Small String Optimization): inline buffer for ≤15 chars
  Layout: {char* ptr, size_t size, union{size_t cap, char buf[16]}}

std::vector<T>:
  {T* begin, T* end, T* capacity_end}

std::map<K,V>:
  Red-black tree: each node has {left, right, parent, color, key, value}

std::unordered_map<K,V>:
  Hash table: {bucket_array, size, load_factor_max, ...}

---

.pyc PEP-552 Magic-Header Forgery (source: pwn.college AoP 2025 day 9)

Trigger: loader validates the first 16 bytes of a .pyc (magic + timestamp + source-hash tag) but doesn't verify the bytecode body. Signals: custom importlib.abc.Loader that checks data[:16]; no hashlib call on data[16:]. Mechanic: Python 3.7+ PEP-552 pyc layout: 4B magic | 4B flags | 8B (timestamp+size OR hash). Prepend the expected header verbatim, append attacker bytecode, loader accepts. Trivial template:

import importlib.util, marshal
magic = importlib.util.MAGIC_NUMBER
with open('out.pyc','wb') as f:
    f.write(magic + b'\x00\x00\x00\x00' + b'\x00'*8 + marshal.dumps(my_code))

Go Interface/itab Restore via GoReSym (source: HTB University 2025 Starshard Reassembly)

Trigger: Go binary with interface-typed method calls (itab tables); standard strings/objdump yield little; Go runtime typelinks intact. Signals: runtime.itab symbol present; go.funcinfo.* section; dispatch through [itab+OFF]. Mechanic: run GoReSym or go-symbol-restore — recovers the typelinks table and resolves each itab to its concrete type + method set. Feed into Ghidra with type propagation to see virtual dispatch as plain calls. Pattern applies to any stripped Go 1.18+ binary.

eBPF kprobe FSM Gated by Syscall Sequence (source: pwn.college AoP 2025 day 4)

Trigger: eBPF attached to a kprobe that mutates a BPF_MAP_TYPE_HASH based on syscall arg hashes; flag only releases when map reaches a specific state. Signals: bpftool prog list non-standard entry; bpftool prog dump xlated id N shows state-machine transitions. Mechanic: see ctf-pwn/sandbox-escape.md cross-ref for the same technique from the pwn angle. For RE: lift bytecode via angr's bpf-ir loader, symbolic-execute to find a sequence of (syscall, arg) tuples reaching accept state.

---

CTF Reverse - Language & Platform-Specific Techniques

Table of Contents

• Python Bytecode Reversing (dis.dis output)
  • Common Pattern: XOR Validation with Split Indices
  • Bytecode Analysis Tips
• Python Opcode Remapping
  • Identification
  • Recovery
• Pyarmor 8/9 Static Unpack (1shot)
• DOS Stub Analysis
• Unity IL2CPP Games
• HarmonyOS HAP/ABC Reverse (abc-decompiler)
• Brainfuck/Esolangs
• UEFI Binary Analysis
• Transpilation to C
• Code Coverage Side-Channel Attack
• Functional Language Reversing (OPAL)
• Python Version-Specific Bytecode (VuwCTF 2025)
• Non-Bijective Substitution Cipher Reversing
• Roblox Place File Analysis
• Godot Game Asset Extraction
• Rust serde_json Schema Recovery
• Android JNI RegisterNatives Obfuscation (HTB WonderSMS)
• Verilog/Hardware Reverse Engineering (srdnlenCTF 2026)
• Prefix-by-Prefix Hash Reversal (Nullcon 2026)
• Ruby/Perl Polyglot Constraint Satisfaction (BearCatCTF 2026)
• Electron App + Native Binary Reversing (RootAccess2026)
• Node.js npm Package Runtime Introspection (RootAccess2026)

For Go and Rust binary reversing, see languages-compiled.md.

---

Python Bytecode Reversing (dis.dis output)

Common Pattern: XOR Validation with Split Indices

Challenge gives raw CPython bytecode (dis.dis disassembly). Common pattern:

1. Check flag length
2. XOR chars at even indices with key1, compare to list p1
3. XOR chars at odd indices with key2, compare to list p2

Reversing:

# Given: p1, p2 (expected values), key1, key2 (XOR keys)
flag = [''] * flag_length
for i in range(len(p1)):
    flag[2*i] = chr(p1[i] ^ key1)      # Even indices
    flag[2*i+1] = chr(p2[i] ^ key2)    # Odd indices
print(''.join(flag))

Bytecode Analysis Tips

• LOAD_CONST followed by COMPARE_OP reveals expected values
• BINARY_XOR identifies the transformation
• BUILD_TUPLE/BUILD_LIST with constants = expected output array
• Loop structure: FOR_ITER + BINARY_SUBSCR = iterating over flag chars
• CALL_FUNCTION on ord = character-to-int conversion

---

Python Opcode Remapping

Identification

Decompiler fails with opcode errors.

Recovery

1. Find modified opcode.pyc in PyInstaller bundle
2. Compare with original Python opcodes
3. Build mapping: {new_opcode: original_opcode}
4. Patch target .pyc
5. Decompile normally

Shortcut (Hack.lu CTF 2013): If the challenge bundles its own modified Python interpreter (e.g., a custom ./py binary), install uncompyle2/uncompyle6 into that interpreter's environment and decompile using the challenge's own runtime. The modified interpreter understands its own opcode mapping, so standard decompilation tools work without manual opcode recovery.

---

Pyarmor 8/9 Static Unpack (1shot)

• Tool: Lil-House/Pyarmor-Static-Unpack-1shot
• Use for Pyarmor 8.x/9.x armored scripts without executing sample code
• Quick signature check: payload typically starts with PY + six digits (Pyarmor 7 and earlier PYARMOR format is not supported)

Workflow:

1. Ensure target directory contains armored scripts and matching pyarmor_runtime library.
2. Run one-shot unpack to emit .1shot. outputs (disassembly + experimental decompile).
3. Treat disassembly as ground truth; verify decompiled source with bytecode when inconsistent.

python /path/to/oneshot/shot.py /path/to/scripts

Optional flags:

# Specify runtime explicitly
python /path/to/oneshot/shot.py /path/to/scripts -r /path/to/pyarmor_runtime.so

# Write outputs to another directory
python /path/to/oneshot/shot.py /path/to/scripts -o /path/to/output

Notes:

• oneshot/pyarmor-1shot executable must exist before running shot.py.
• PyInstaller bundles or archives should be unpacked first, then processed with 1shot.

---

DOS Stub Analysis

PE files can hide code in DOS stub:

1. Check for large DOS stub in Ghidra/IDA
2. Run in DOSBox
3. Load in IDA as 16-bit DOS
4. Look for int 16h (keyboard input)

---

Unity IL2CPP Games

• Use Il2CppDumper to dump symbols
• If Il2CppDumper fails, consider that global-metadata.dat may be encrypted; search strings/xrefs in the main binary and inspect the metadata loading path for custom decryption before dump.
• Look for Start() functions
• Key derivation: key = SHA256(companyName + "\n" + productName)
• Decrypt server responses with derived key

Please note most of that the executable file for the PC platform is GameAssembly.dll or *Assembly.dll, for the Android is libil2cpp.so.

---

HarmonyOS HAP/ABC Reverse (abc-decompiler)

• Target files: .hap package and embedded .abc bytecode
• Tool: https://github.com/ohos-decompiler/abc-decompiler
• Download jadx-dev-all.jar from releases

Critical startup note:

• java -jar may enter GUI mode
• For CLI mode, always use:

java -cp "./jadx-dev-all.jar" jadx.cli.JadxCLI [options] <input>

Most common commands:

# Basic decompile to directory
java -cp "./jadx-dev-all.jar" jadx.cli.JadxCLI -d "out" ".abc"

# Decompile .abc (recommended for this scenario)
java -cp "./jadx-dev-all.jar" jadx.cli.JadxCLI -m simple -d "out_hap" "modules.abc"

Recommended parameters for this challenge:

• -m simple: reduce high-level reconstruction to avoid SSA/PHI-heavy failures
• --log-level ERROR: keep only critical errors
• Full recommended command:

java -cp "./jadx-dev-all.jar" jadx.cli.JadxCLI -m simple --log-level ERROR -d "out_abc_simple" "modules.abc"

Parameter quick reference:

• -d output directory
• --help help

Notes:

• .hap is a package: extract it first (zip), then locate and analyze .abc
• Quote paths containing spaces or non-ASCII characters
• Use a new output directory name per run to avoid stale results
• Errors do not always mean full failure; prioritize out_xxx/sources/
• If auto fails, switch to -m simple first

Standard workflow:

1. Run with -m simple --log-level ERROR
2. Inspect key business files in output (for example pages/Index.java)
3. If cleaner output is needed, retry with -m auto or -m restructure
4. If some methods still fail, keep the simple output and continue logic analysis via alternate paths

---

Brainfuck/Esolangs

• Check if compiled with known tools (BF-it)
• Understand tape/memory model
• Static analysis of cell operations

---

UEFI Binary Analysis

7z x firmware.bin -oextracted/
file extracted/* | grep "PE32+"

• Bootkit replaces boot loader
• Custom VM protects decryption
• Lift VM bytecode to C

---

Transpilation to C

For heavily obfuscated code:

for opcode, args in instructions:
    if opcode == 'XOR':
        print(f"r{args[0]} ^= r{args[1]};")
    elif opcode == 'ADD':
        print(f"r{args[0]} += r{args[1]};")

Compile with -O3 for constant folding.

---

Code Coverage Side-Channel Attack

Pattern (Coverup, Nullcon 2026): PHP challenge provides XDebug code coverage data alongside encrypted output.

How it works:

• PHP code uses xdebug_start_code_coverage(XDEBUG_CC_UNUSED | XDEBUG_CC_DEAD_CODE | XDEBUG_CC_BRANCH_CHECK)
• Encryption uses data-dependent branches: if ($xored == chr(0)) ... if ($xored == chr(1)) ...
• Coverage JSON reveals which branches were executed during encryption
• This leaks the set of XOR intermediate values that occurred

Exploitation:

import json

# Load coverage data
with open('coverage.json') as f:
    cov = json.load(f)

# Extract executed XOR values from branch coverage
executed_xored = set()
for line_no, hit_count in cov['encrypt.php']['lines'].items():
    if hit_count > 0:
        # Map line numbers to the chr(N) value in the if-statement
        executed_xored.add(extract_value_from_line(line_no))

# For each position, filter candidates
for pos in range(len(ciphertext)):
    candidates = []
    for key_byte in range(256):
        xored = plaintext_byte ^ key_byte  # or reverse S-box lookup
        if xored in executed_xored:
            candidates.append(key_byte)
    # Combined with known plaintext prefix, this uniquely determines key

Key insight: Code coverage is a powerful oracle — it tells you which conditional paths were taken. Any encryption with data-dependent branching leaks information through coverage.

Mitigation detection: Look for branchless/constant-time crypto implementations that defeat this attack.

---

Functional Language Reversing (OPAL)

Pattern (Opalist, Nullcon 2026): Binary compiled from OPAL (Optimized Applicative Language), a purely functional language.

Recognition markers:

• .impl (implementation) and .sign (signature) source files
• IMPLEMENTATION / SIGNATURE keywords
• Nested IF..THEN..ELSE..FI structures
• Functions named f1, f2, ... fN (numeric naming)
• Heavy use of seq[nat], string, denotation types

Reversing approach:

1. Pure functions are mathematically invertible — reverse each step in the pipeline
2. Identify the transformation chain: f_final(f_n(...f_2(f_1(input))...))
3. For each function, build the inverse

Aggregate brute-force for scramble functions: When a transformation accumulates state that depends on original (unknown) values:

# Example: f8 adds cumulative offset based on parity of original bytes
# offset contribution per element depends on whether pre-scramble value is even/odd
# Total offset S = sum of contributions, but S mod 256 has only 256 possibilities

decoded = base64_decode(target)
for total_offset_S in range(256):
    candidate = [(b - total_offset_S) % 256 for b in decoded]
    # Verify: recompute S from candidate values
    recomputed_S = sum(contribution(i, candidate[i]) for i in range(len(candidate))) % 256
    if recomputed_S == total_offset_S:
        # Apply remaining inverse steps
        result = apply_inverse_substitution(candidate)
        if all(32 <= c < 127 for c in result):
            print(bytes(result))

Key lesson: When a scramble function has a chicken-and-egg dependency (result depends on original, which is unknown), brute-force the aggregate effect (often mod 256 = 256 possibilities) rather than all possible states (exponential).

---

Python Version-Specific Bytecode (VuwCTF 2025)

Pattern (A New Machine): Challenge targets specific Python version (e.g., 3.14.0 alpha).

Key requirement: Compile that exact Python version to disassemble bytecode — alpha/beta versions have different opcodes than stable releases.

# Build specific Python version
wget https://www.python.org/ftp/python/3.14.0/Python-3.14.0a4.tar.xz
tar xf Python-3.14.0a4.tar.xz
cd Python-3.14.0a4 && ./configure && make -j$(nproc)
./python -c "import dis, marshal; dis.dis(marshal.loads(open('challenge.pyc','rb').read()[16:]))"

Common validation: Flag compared against tuple of squared ASCII values:

# Reverse: flag[i] = sqrt(expected_tuple[i])
import math
flag = ''.join(chr(int(math.isqrt(v))) for v in expected_values)

---

Non-Bijective Substitution Cipher Reversing

Pattern (Coverup, Nullcon 2026): S-box/substitution table has collisions (multiple inputs map to same output).

Detection:

sbox = [...]  # substitution table
if len(set(sbox)) < len(sbox):
    print("Non-bijective! Collisions exist.")

Building reverse lookup:

from collections import defaultdict
rev_sub = defaultdict(list)
for i, v in enumerate(sbox):
    rev_sub[v].append(i)
# rev_sub[output] = [list of possible inputs]

Disambiguation strategies:

1. Known plaintext format (e.g., ENO{, flag{) fixes key bytes at known positions
2. Side-channel data (code coverage, timing) eliminates impossible candidates
3. Printable ASCII constraint (32-126) reduces candidate space
4. Re-encrypt candidates and verify against known ciphertext

---

Roblox Place File Analysis

Pattern (MazeRunna, 0xFun 2026): Roblox game with flag hidden in older version; latest version contains decoy.

Version history via Asset Delivery API:

# Extract placeId and universeId from game page HTML
# Query each version (requires .ROBLOSECURITY cookie):
curl -H "Cookie: .ROBLOSECURITY=..." \
  "https://assetdelivery.roblox.com/v2/assetId/{placeId}/version/1"
# Download location URL → place_v1.rbxlbin

Binary format parsing: .rbxlbin files contain chunks:

• INST — class buckets and referent IDs
• PROP — per-instance fields (including Script.Source)
• PRNT — parent-child relationships (object tree)

Decode chunk payloads, walk PROP entries for Source field, dump Script.Source / LocalScript.Source per version, then diff.

Key lesson: Always check version history. Latest version may contain decoy flag while real flag is in an older version. Diff script sources across versions.

---

Godot Game Asset Extraction

Pattern (Steal the Xmas): Encrypted Godot .pck packages.

Tools:

• gdsdecomp - Extract Godot packages
• KeyDot - Extract encryption key from Godot executables

Workflow:

1. Run KeyDot against game executable → extract encryption key
2. Input key into gdsdecomp
3. Extract and open project in Godot editor
4. Search scripts/resources for flag data

---

Rust serde_json Schema Recovery

Pattern (Curly Crab, PascalCTF 2026): Rust binary reads JSON from stdin, deserializes via serde_json, prints success/failure emoji.

Approach:

1. Disassemble serde-generated Visitor implementations
2. Each visitor's visit_map / visit_seq reveals expected keys and types
3. Look for string literals in deserializer code (field names like "pascal", "CTF")
4. Reconstruct nested JSON schema from visitor call hierarchy
5. Identify value types from visitor method names: visit_str = string, visit_u64 = number, visit_bool = boolean, visit_seq = array

{"pascal":"CTF","CTF":2026,"crab":{"I_":true,"cr4bs":1337,"crabby":{"l0v3_":["rust"],"r3vv1ng_":42}}}

Key insight: Flag is the concatenation of JSON keys in schema order. Reading field names in order reveals the flag.

---

Verilog/Hardware Reverse Engineering (srdnlenCTF 2026)

Pattern (Rev Juice): Verilog HDL source for a vending machine with hidden product unlocked by specific coin insertion and selection sequence.

Approach:

1. Analyze Verilog modules to understand state machine and history tracking
2. Identify hidden conditions (e.g., product 8 enabled only when COINS_HISTORY array has specific values at specific taps)
3. Build timing model for each action type (how many clock cycles each operation takes)
4. Work backward from required history values to construct the correct input sequence

Timing model construction:

# Map each action to its cycle count (determined from Verilog state machines)
TIMING = {
    "insert_coin": 3,       # 3 cycles per coin insertion
    "select_success": 7,    # 7 cycles for successful product selection
    "select_fail": 5,       # 5 cycles for failed selection attempt
    "cancel_with_coins": 4, # 4 cycles for cancel when coins > 0
    "cancel_at_zero": 2,    # 2 cycles for cancel when coins = 0
}

# COINS_HISTORY is a shift register updated each cycle
# History tap requirements (from Verilog conditions):
# H[0]=1, H[7]=4, H[28]=H[33]=H[38]=6
# H[63]=H[73]=2, H[80]=9
# (H[19]+H[21]+H[56]+H[69]) mod 32 = 0

Key insight: Hardware challenges require understanding the exact timing model — each operation takes a specific number of clock cycles, and shift registers record history at fixed tap positions. Work backward from the required tap values to determine what action must have occurred at each cycle. The solution is often a specific sequence notation (e.g., I9C_SP6_CNL_I2C_SP2_I6C_SP6_SP6_SP5_CNL_I4C_SP1).

Detection: Look for .v or .sv (Verilog/SystemVerilog) files, always @(posedge clk) blocks, shift register patterns, and state machine case statements with hidden conditions gated on history values.

---

Prefix-by-Prefix Hash Reversal (Nullcon 2026)

See patterns-ctf-2.md for the full technique. This section covers language-specific considerations.

Language-specific notes:

• Hash algorithm may be uncommon (MD2, custom) — don't need to identify it, just match outputs by running the binary
• Use subprocess.run() with timeout=2 to handle binaries that hang on bad input
• For stripped binaries, check if ltrace reveals the hash function name (e.g., MD2_Update)

---

Android JNI RegisterNatives Obfuscation (HTB WonderSMS)

Pattern: Android app loads native library with System.loadLibrary(), but uses RegisterNatives in JNI_OnLoad instead of standard JNI naming convention (Java_com_pkg_Class_method). This hides which C++ function handles each Java native method.

Identification:

// In decompiled Java (jadx):
static { System.loadLibrary("audio"); }
private final native ProcessedMessage processMessage(SmsMessage msg);

Standard JNI would have a symbol Java_com_rloura_wondersms_SmsReceiver_processMessage. If that symbol is missing from the .so, RegisterNatives is being used.

Finding the real handler in Ghidra:

1. Locate JNI_OnLoad (exported symbol, always present)
2. Trace to RegisterNatives(env, clazz, methods, count) call
3. The methods array contains {name, signature, fnPtr} structs
4. Follow fnPtr to find the actual native function

// JNI_OnLoad registers functions manually:
static JNINativeMethod methods[] = {
    {"processMessage", "(Landroid/telephony/SmsMessage;)LProcessedMessage;", (void*)real_handler}
};
(*env)->RegisterNatives(env, clazz, methods, 1);

Architecture selection for analysis:

# x86_64 gives best Ghidra decompilation (most similar to desktop code)
# Extract from APK:
unzip WonderSMS.apk -d extracted/
ls extracted/lib/x86_64/  # Prefer this over arm64-v8a for static analysis

Key insight: RegisterNatives is a deliberate obfuscation technique — it decouples Java method names from native symbol names, making it impossible to find handlers by string search alone. Always check JNI_OnLoad first when reversing Android native libraries with stripped symbols.

Detection: Native method declared in Java + no matching JNI symbol in .so + JNI_OnLoad present. The library is typically stripped (no debug symbols).

---

Ruby/Perl Polyglot Constraint Satisfaction (BearCatCTF 2026)

Pattern (Polly's Key): A single file valid in both Ruby and Perl. Each language imposes different validation constraints on a 50-character key. Satisfy both simultaneously to decrypt the flag.

Polyglot structure exploits:

• Ruby: =begin...=end is a block comment
• Perl: =begin...=cut is POD (Plain Old Documentation), =end is ignored
• Different code runs in each language based on comment block boundaries

Typical constraints:

• Ruby: Character set must form a mathematical property (e.g., all 50 printable ASCII chars except ^ used exactly once, each satisfying XOR(val, (val-16) % 257) is a primitive root mod 257)
• Perl: Ordering constraint via insertion sort inversion count (hardcoded inversion table determines exact permutation)

Solution approach:

1. Find the valid character set (mathematical constraint from one language)
2. Use the ordering constraint (from other language) to determine exact arrangement
3. Compute key hash (e.g., MD5) and decrypt

# Determine character ordering from inversion counts
def reconstruct_from_inversions(chars, inv_counts):
    result = []
    remaining = sorted(chars)
    for i in range(len(chars) - 1, -1, -1):
        # inv_counts[i] = number of elements to the left that are greater
        idx = inv_counts[i]
        result.insert(idx, remaining.pop(i))
    return result

Key insight: Polyglot files exploit language-specific comment/block syntax to run different code in each interpreter. The constraints from both languages intersect to uniquely determine the key. Identify which code runs in which language by testing the file with both interpreters and comparing behavior.

Detection: File that runs under multiple interpreters (ruby file && perl file). Challenge mentions "polyglot" or provides a file ending in .rb that also looks like Perl.

---

Electron App + Native Binary Reversing (RootAccess2026)

Pattern (Rootium Browser): Electron desktop app bundles a native ELF/DLL binary for sensitive operations (vault, crypto, auth). The Electron layer is a wrapper; the real flag logic is in the native binary.

Extraction workflow:

1. Unpack Electron ASAR archive:

# Install ASAR tool
npm install -g @electron/asar

# Extract the app.asar archive
asar extract resources/app.asar app_extracted/
ls app_extracted/

2. Locate native binary: Search for ELF/DLL files called from JavaScript:

# Find native binaries
find app_extracted/ -name "*.node" -o -name "*.so" -o -name "*vault*" -o -name "*auth*"

# Check JS for child_process.spawn or ffi-napi calls
grep -r "spawn\|execFile\|ffi\|require.*native" app_extracted/

3. Reverse the native binary (XOR + rotation cipher example):

def decrypt_password(encrypted_bytes, key):
    """Common pattern: XOR with constant + bit rotation + key XOR."""
    result = []
    for i, byte in enumerate(encrypted_bytes):
        decrypted = ((byte ^ 0x42) >> 3) ^ key[i % len(key)]
        result.append(chr(decrypted))
    return ''.join(result)

def decrypt_flag(encrypted_flag, password):
    """Flag uses password as key with position-dependent rotation."""
    result = []
    for i, byte in enumerate(encrypted_flag):
        key_byte = ord(password[i % len(password)])
        decrypted = ((byte ^ 0x7E) >> (i % 8)) ^ key_byte
        result.append(chr(decrypted))
    return ''.join(result)

Key insight: Electron apps are JavaScript wrapping native code. Extract with asar, then focus on the native binary. The JS layer often contains the password verification flow in plaintext, revealing what the native binary expects. Look for encrypted data in the .data or .rodata sections of the ELF.

Detection: .asar files in resources/ directory, Electron framework files, package.json with electron dependency.

---

Node.js npm Package Runtime Introspection (RootAccess2026)

Pattern (RootAccess CLI): Obfuscated npm package with RC4 encoding, control flow flattening, and flag split across multiple fragments. Static analysis is impractical — use runtime introspection instead.

Dynamic analysis approach:

#!/usr/bin/env node

// 1. Load obfuscated modules
const cryptoMod = require('target-package/dist/lib/crypto.js');
const vaultMod = require('target-package/dist/lib/vault.js');

// 2. Enumerate all exported properties
for (const mod of [cryptoMod, vaultMod]) {
    for (const key of Object.keys(mod)) {
        const obj = mod[key];
        console.log(`Export: ${key}`);
        // List all methods including hidden ones
        const props = Object.getOwnPropertyNames(obj);
        const proto = Object.getOwnPropertyNames(obj.prototype || {});
        console.log('  Own:', props);
        console.log('  Proto:', proto);
    }
}

// 3. Extract flag fragments
const Engine = cryptoMod.CryptoEngine;
const total = Engine.getTotalFragments();
let flag = '';
for (let i = 1; i <= total; i++) {
    flag += Engine.getFragment(i);
}
console.log('Flag:', flag);

// 4. Check for hidden methods (common: __getFullFlag__, _debug, _raw)
const hidden = Object.getOwnPropertyNames(Engine)
    .filter(p => p.startsWith('__') || p.startsWith('_'));
console.log('Hidden methods:', hidden);

Key insight: Heavily obfuscated JavaScript (control flow flattening, RC4 string encoding, dead code) makes static analysis prohibitively slow. Runtime introspection via Object.getOwnPropertyNames() reveals all methods including hidden ones. The module's own decryption runs automatically when loaded — just call the decoded functions directly.

Detection: npm package with minified/obfuscated dist/ directory, challenge says "reverse engineer the CLI tool", package.json with custom commands.

---

CTF Reverse - Competition-Specific Patterns (Part 2)

Table of Contents

• Multi-Layer Self-Decrypting Binary (DiceCTF 2026)
• Embedded ZIP + XOR License Decryption (MetaCTF 2026)
• Stack String Deobfuscation from .rodata XOR Blob (Nullcon 2026)
• Prefix Hash Brute-Force (Nullcon 2026)
• CVP/LLL Lattice for Constrained Integer Validation (HTB ShadowLabyrinth)
• Decision Tree Function Obfuscation (HTB WonderSMS)
• GLSL Shader VM with Self-Modifying Code (ApoorvCTF 2026)
• GF(2^8) Gaussian Elimination for Flag Recovery (ApoorvCTF 2026)
• Z3 for Single-Line Python Boolean Circuit (BearCatCTF 2026)
• Sliding Window Popcount Differential Propagation (BearCatCTF 2026)
• Morse Code from Keyboard LEDs via ioctl (PlaidCTF 2013)
• Binary Repurposed as Keystream Oracle (DEF CON 33 Quals 2025)
• Unicorn Host/Guest Memory Divergence — see ctf-pwn

---

Multi-Layer Self-Decrypting Binary (DiceCTF 2026)

Pattern (another-onion): Binary with N layers (e.g., 256), each reading 2 key bytes, deriving keystream via SHA-256 NI instructions, XOR-decrypting the next layer, then jumping to it. Must solve within a time limit (e.g., 30 minutes).

Oracle for correct key: Wrong key bytes produce garbage code. Correct key bytes produce code with exactly 2 call read@plt instructions (next layer's reads). Brute-force all 65536 candidates per layer using this oracle.

JIT execution approach (fastest):

// Map binary's memory at original virtual addresses into solver process
// Compile solver at non-overlapping address: -Wl,-Ttext-segment=0x10000000
void *text = mmap((void*)0x400000, text_size, PROT_RWX, MAP_FIXED|MAP_PRIVATE, fd, 0);
void *bss = mmap((void*)bss_addr, bss_size, PROT_RW, MAP_FIXED|MAP_SHARED, shm_fd, 0);

// Patch read@plt to inject candidate bytes instead of reading stdin
// Patch tail jmp/call to next layer with ret/NOP to return from layer

// Fork-per-candidate: COW gives isolated memory without memcpy
for (int candidate = 0; candidate < 65536; candidate++) {
    pid_t pid = fork();
    if (pid == 0) {
        // Child: remap BSS as MAP_PRIVATE (COW from shared file)
        mmap(bss_addr, bss_size, PROT_RW, MAP_FIXED|MAP_PRIVATE, shm_fd, 0);
        inject_key(candidate >> 8, candidate & 0xff);
        ((void(*)())layer_addr)();  // Execute layer as function call
        // Check: does decrypted code contain exactly 2 call read@plt?
        if (count_read_calls(next_layer_addr) == 2) signal_found(candidate);
        _exit(0);
    }
}

Performance tiers:

Approach	Speed	256-layer estimate
Python subprocess	~2/s	days
Ptrace fork injection	~119/s	6+ hours
JIT + fork-per-candidate	~1000/s	140 min
JIT + shared BSS + 32 workers	~3500/s	~17 min

Shared BSS optimization: BSS (16MB+) stored in /dev/shm as MAP_SHARED in parent. Children remap as MAP_PRIVATE for COW. Reduces fork overhead from 16MB page-table setup to ~4KB.

Key insight: Multi-layer decryption challenges are fundamentally about building fast brute-force engines. JIT execution (mapping binary memory into solver, running code directly as function calls) is orders of magnitude faster than ptrace. Fork-based COW provides free memory isolation per candidate.

Gotchas:

• Real binary may use call (0xe8) instead of jmp (0xe9) for layer transitions — adjust tail patching
• BSS may extend beyond ELF MemSiz via kernel brk mapping — map extra space
• SHA-NI instructions work even when not advertised in /proc/cpuinfo

---

Embedded ZIP + XOR License Decryption (MetaCTF 2026)

Pattern (License To Rev): Binary requires a license file as argument. Contains an embedded ZIP archive with the expected license, and an XOR-encrypted flag.

Recognition:

• strings reveals EMBEDDED_ZIP and ENCRYPTED_MESSAGE symbols
• Binary is not stripped — nm or readelf -s shows data symbols in .rodata
• file shows PIE executable, source file named licensed.c

Analysis workflow:

1. Find data symbols:

readelf -s binary | grep -E "EMBEDDED|ENCRYPTED|LICENSE"
# EMBEDDED_ZIP at offset 0x2220, 384 bytes
# ENCRYPTED_MESSAGE at offset 0x21e0, 35 bytes

2. Extract embedded ZIP:

import struct
with open('binary', 'rb') as f:
    data = f.read()
# Find PK\x03\x04 magic in .rodata
zip_start = data.find(b'PK\x03\x04')
# Extract ZIP (size from symbol table or until next symbol)
open('embedded.zip', 'wb').write(data[zip_start:zip_start+384])

3. Extract license from ZIP:

unzip embedded.zip  # Contains license.txt

4. XOR decrypt the flag:

license = open('license.txt', 'rb').read()
enc_msg = open('encrypted_msg.bin', 'rb').read()  # Extract from .rodata
flag = bytes(a ^ b for a, b in zip(enc_msg, license))
print(flag.decode())

Key insight: No need to run the binary or bypass the expiry date check. The embedded ZIP and encrypted message are both in .rodata — extract and XOR offline.

Disassembly confirms:

• memcmp(user_license, decompressed_embedded_zip, size) — license validation
• Date parsing with sscanf("%d-%d-%d") on EXPIRY_DATE= field
• XOR loop: ENCRYPTED_MESSAGE[i] ^ license[i] → putc() per byte

Lesson: When a binary has named symbols (EMBEDDED_*, ENCRYPTED_*), extract data directly from the binary without execution. XOR with known plaintext (the license) is trivially reversible.

---

Stack String Deobfuscation from .rodata XOR Blob (Nullcon 2026)

Pattern (stack_strings_1/2): Binary mmaps a blob from .rodata, XOR-deobfuscates it, then uses the blob to validate input. Flag is recovered by reimplementing the verification loop.

Recognition:

• mmap() call followed by XOR loop over .rodata data
• Verification loop with running state (eax, ebx, r9) updated with constants like 0x9E3779B9, 0x85EBCA6B, 0xA97288ED
• rol32() operations with position-dependent shifts
• Expected bytes stored in deobfuscated buffer

Approach:

1. Extract .rodata blob with pyelftools:

   from elftools.elf.elffile import ELFFile
   with open(binary, "rb") as f:
       elf = ELFFile(f)
       ro = elf.get_section_by_name(".rodata")
       blob = ro.data()[offset:offset+size]
   

2. Recover embedded constants (length, magic values) by XOR with known keys from disassembly
3. Reimplement the byte-by-byte verification loop:
   • Each iteration: compute two hash-like values from running state
   • XOR them together and with expected byte to recover input byte
   • Update running state with constant additions

Variant (stack_strings_2): Adds position permutation + state dependency on previous character:

• Position permutation: byte i may go to position pos[i] in the output
• State dependency: need = (expected - rol8(prev_char, 1)) & 0xFF
• Must track state variable that updates to current character each iteration

Key constants to look for:

• 0x9E3779B9 (golden ratio fractional, common in hash functions)
• 0x85EBCA6B (MurmurHash3 finalizer constant)
• 0xA97288ED (related hash constant)
• rol32() with shift i & 7

---

Prefix Hash Brute-Force (Nullcon 2026)

Pattern (Hashinator): Binary hashes every prefix of the input independently and outputs one digest per prefix. Given N output digests, the flag has N-1 characters.

Attack: Recover input one character at a time:

for pos in range(1, len(target_hashes)):
    for ch in charset:
        candidate = known_prefix + ch + padding
        hashes = run_binary(candidate)
        if hashes[pos] == target_hashes[pos]:
            known_prefix += ch
            break

Key insight: If each prefix hash is independent (no chaining/HMAC), the problem decomposes into N x |charset| binary executions. This is the hash equivalent of byte-at-a-time block cipher attacks.

Detection: Binary outputs multiple hash lines. Changing last character only changes last hash. Different input lengths produce different numbers of output lines.

---

CVP/LLL Lattice for Constrained Integer Validation (HTB ShadowLabyrinth)

Pattern: Binary validates flag via matrix multiplication where grouped input characters are multiplied by coefficient matrices and checked against expected 64-bit results. Standard algebra fails because solutions must be printable ASCII (32-126). Lattice-based CVP (Closest Vector Problem) with LLL reduction solves this efficiently.

Identification:

1. Binary groups input characters (e.g., 4 at a time)
2. Each group is multiplied by a coefficient matrix
3. Results compared against hardcoded 64-bit values
4. Need integer solutions in a constrained range (printable ASCII)

SageMath CVP solver:

from sage.all import *

def solve_constrained_matrix(coefficients, targets, char_range=(32, 126)):
    """
    coefficients: list of coefficient rows (e.g., 4 values per group)
    targets: expected output values
    char_range: valid character range (printable ASCII)
    """
    n = len(coefficients[0])  # characters per group
    mid = (char_range[0] + char_range[1]) // 2

    # Build lattice: [coeff_matrix | I*scale]
    # The target vector includes adjusted targets
    M = matrix(ZZ, n + len(targets), n + len(targets))
    scale = 1000  # Weight to constrain character range

    for i, row in enumerate(coefficients):
        for j, c in enumerate(row):
            M[j, i] = c
        M[n + i, i] = 1  # padding

    for j in range(n):
        M[j, len(targets) + j] = scale

    target_vec = vector(ZZ, [t - sum(c * mid for c in row)
                              for row, t in zip(coefficients, targets)]
                        + [0] * n)

    # LLL + CVP
    L = M.LLL()
    closest = L * L.solve_left(target_vec)  # or use Babai
    solution = [closest[len(targets) + j] // scale + mid for j in range(n)]
    return bytes(solution)

Two-phase validation pattern:

1. Phase 1 (matrix math): Solve via CVP/LLL → recovers first N characters
2. First N characters become AES key → decrypt file.bin (XOR last 16 bytes + AES-256-CBC + zlib decompress)
3. Phase 2 (custom VM): Decrypted bytecode runs in custom VM, validates remaining characters via another linear system (mod 2^32)

Modular linear system solving (Phase 2 — VM validation):

import numpy as np
from sympy import Matrix

# M * x = v (mod 2^32)
M_mod = Matrix(coefficients) % (2**32)
v_mod = Matrix(targets) % (2**32)
# Gaussian elimination in Z/(2^32)
solution = M_mod.solve(v_mod)  # Returns flag characters

Key insight: When a binary validates input through linear combinations with large coefficients and the solution must be in a small range (printable ASCII), this is a lattice problem in disguise. LLL reduction + CVP finds the nearest lattice point, recovering the constrained solution. Cross-reference: invoke /ctf-crypto for LLL/CVP fundamentals (advanced-math.md in ctf-crypto).

Detection: Binary performs matrix-like operations on grouped input, compares against 64-bit constants, and a brute-force search space is too large (e.g., 256^4 per group × 12 groups).

---

Decision Tree Function Obfuscation (HTB WonderSMS)

Pattern: Binary routes input through ~200+ auto-generated functions, each computing a polynomial expression from input positions, comparing against a constant, and branching left/right. The tree makes static analysis impractical without scripted extraction.

Identification:

1. Large number of similar functions with random-looking names (e.g., f315732804)
2. Each function computes arithmetic on specific input positions
3. Functions call other tree functions or a final validation function
4. Decompiled code shows if (expr cmp constant) call_left() else call_right()

Ghidra headless scripting for mass extraction:

# Extract comparison constants from all tree functions
# Run via: analyzeHeadless project/ tmp -import binary -postScript extract_tree.py
from ghidra.program.model.listing import *
from ghidra.program.model.symbol import *

fm = currentProgram.getFunctionManager()
results = []
for func in fm.getFunctions(True):
    name = func.getName()
    if name.startswith('f') and name[1:].isdigit():
        # Find CMP instruction and extract immediate constant
        inst_iter = currentProgram.getListing().getInstructions(func.getBody(), True)
        for inst in inst_iter:
            if inst.getMnemonicString() == 'CMP':
                operand = inst.getOpObjects(1)
                if operand:
                    results.append((name, int(operand[0].getValue())))

Constraint propagation from known output format:

1. Start from known output bytes (e.g., http://HTB{...}) → fix several input positions
2. Fixed positions cascade through arithmetic constraints → determine dependent positions
3. Tree root equation pins down remaining free variables
4. Recognize English words in partial flag to disambiguate multiple solutions

Key insight: Auto-generated decision trees look overwhelming but are repetitive by construction. Script the extraction (Ghidra, Binary Ninja, radare2) rather than reversing each function manually. The tree is just a dispatcher — the real logic is in the leaf function and its constraints.

Detection: Binary with hundreds of similarly-structured functions, 3-5 input position references per function, branching to two other functions or a common leaf.

---

GLSL Shader VM with Self-Modifying Code (ApoorvCTF 2026)

Pattern (Draw Me): A WebGL2 fragment shader implements a Turing-complete VM on a 256x256 RGBA texture. The texture is both program memory and display output.

Texture layout:

• Row 0: Registers (pixel 0 = instruction pointer, pixels 1-32 = general purpose)
• Rows 1-127: Program memory (RGBA = opcode, arg1, arg2, arg3)
• Rows 128-255: VRAM (display output)

Opcodes: NOP(0), SET(1), ADD(2), SUB(3), XOR(4), JMP(5), JNZ(6), VRAM-write(7), STORE(8), LOAD(9). 16 steps per frame.