| name | ctf-malware |
| description | Malware & network analysis: obfuscated JS/PowerShell/VBA, PE/.NET analysis, PyInstaller/PyArmor, custom C2 protocols (RC4, AES, DNS), YARA rules, shellcode, memory-forensics IOCs, anti-sandbox/anti-VM bypass. |
CTF Malware & Network Analysis
Quick reference for malware analysis CTF challenges. Each technique has a one-liner here; see supporting files for full details with code.
Additional Resources
- scripts-and-obfuscation.md - JavaScript deobfuscation, PowerShell analysis, eval/base64 decoding, junk code detection, hex payloads, Debian package analysis, dynamic analysis techniques (strace/ltrace, network monitoring, memory string extraction, automated sandbox execution), YARA rules for malware detection, shellcode analysis (Unicorn Engine, Capstone), memory forensics for malware (Volatility 3 malfind, process injection detection), anti-analysis techniques (VM detection, timing evasion, API hashing, process injection)
- c2-and-protocols.md - C2 traffic patterns, custom crypto protocols, RC4 WebSocket, DNS-based C2, network indicators, PCAP analysis, AES-CBC, encryption ID, Telegram bot recovery, Poison Ivy RAT Camellia decryption
- pe-and-dotnet.md - PE analysis (peframe, pe-sieve, pestudio), .NET analysis (dnSpy, AsmResolver), LimeRAT extraction, sandbox evasion, malware config extraction, PyInstaller+PyArmor
Pattern Recognition Index
Dispatch on byte/file signals, never the sample's reputation.
| Signal | Technique → file |
|---|
File starts with PK + contains package.json + .vsix extension | VSCode extension activation-event exfil → scripts-and-obfuscation.md |
High-entropy extension.js/index.js > 50 KB with child_process | JS obfuscator.io / JSFuck deobfuscation → scripts-and-obfuscation.md |
PowerShell -enc <b64> or IEX in parent process logs | PS script decode + reflection assembly → scripts-and-obfuscation.md |
| PCAP with custom TCP port, payload bytes distributed uniformly | Custom crypto / RC4 / AES session key recovery → c2-and-protocols.md |
PCAP with DNS TXT records carrying base32/64 blobs | DNS-based C2 exfil → c2-and-protocols.md |
| PCAP where specific User-Agent receives non-default responses | UA-gated C2 path-hex-XOR → (see ctf-forensics/network-advanced.md) |
PE32/PE64 with .text sections having packer signatures (UPX, VMProtect, Themida) | Unpacking + dump-after-decrypt → pe-and-dotnet.md |
| .NET assembly with obfuscated strings + reflection calls | dnSpy + de4dot + string decryption → pe-and-dotnet.md |
PyInstaller binary (MEIPASS, _MEIxxxx strings) | pyinstxtractor + pycdc → pe-and-dotnet.md |
Process-injection IoCs in memory dump (RWX allocations, NtUnmapViewOfSection) | Volatility malfind + process hollowing detection → scripts-and-obfuscation.md |
Anti-VM checks (cpuid hypervisor bit, RDTSC timing, specific drivers) | VM-detection bypass in analysis sandbox → scripts-and-obfuscation.md |
| WebSocket frame with visible RC4 key-setup or KSA signatures | RC4 WebSocket reversing → c2-and-protocols.md |
Recognize the artefact or protocol pattern. The sample's name is not a signal.
For inline snippets and quick-reference tables, see quickref.md. The Pattern Recognition Index above is the dispatch table — always consult it first.
C2 Traffic and Protocol Analysis
Table of Contents
PCAP Analysis
tshark -r file.pcap -Y "tcp.stream eq X" -T fields -e tcp.payload
Look for C2 communication patterns on unusual ports (e.g., port 21 not for FTP).
Custom Crypto Protocols
- Stream ciphers may share keystream state for both directions
- Concatenate ALL payloads chronologically before decryption
- Look for hardcoded keys in
.rodata
- ChaCha20 keystream extraction: Send large nullbytes payload (0 XOR anything = anything)
- Alternative: Pipe ciphertext from pcap directly into the binary
C2 Traffic Patterns
- Beaconing: regular intervals
- Domain generation algorithms (DGA)
- Encoded/encrypted payloads
- HTTP(S) with custom headers
- DNS tunneling
Network Indicators
strings malware | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
strings malware | grep -E '[a-zA-Z0-9.-]+\.(com|net|org|io)'
tshark -r capture.pcap -Y "dns.qry.name" -T fields -e dns.qry.name | sort -u
RC4-Encrypted WebSocket C2 Traffic
Pattern (Tampered Seal): Malware uses WSS over non-standard port with RC4 encryption.
Decryption workflow:
- Identify C2 port from malware source (not standard 443)
- Remap port with
tcprewrite so Wireshark decodes TLS
- Add RSA key for TLS decryption -> reveals WebSocket frames
- Find RC4 key hardcoded in malware binary
- Decrypt each WebSocket payload with RC4 via CyberChef
Malware communication patterns:
- Registration message: hostname, OS, username, privileges
- Exfiltration: screenshots, keylog data, file contents
- Commands: reverse shell, file download, process list
Password Rotation in C2
Pattern: C2 uses rotating passwords based on time/sequence
Analysis:
- Find password generation function
- Identify rotation trigger (time-based, message count)
- Sync your decryptor with the rotation
def get_current_password(timestamp):
hour_bucket = timestamp // 3600
return hashlib.sha256(f"seed_{hour_bucket}".encode()).digest()
AES-CBC in Malware
Common key derivation:
- MD5/SHA256 of hardcoded string
- Derived from timestamp or PID
- Password-based (PBKDF2)
Analysis approach:
from Crypto.Cipher import AES
from Crypto.Util.Padding import unpad
import hashlib
password = b"hardcoded_password"
key = hashlib.md5(password).digest()
iv = ciphertext[:16]
ct = ciphertext[16:]
cipher = AES.new(key, AES.MODE_CBC, iv)
plaintext = unpad(cipher.decrypt(ct), 16)
Identifying Encryption Algorithms
By constants:
- AES:
0x637c777b, 0x63636363 (S-box)
- ChaCha20:
expand 32-byte k or 0x61707865
- RC4: Sequential S-box initialization
- TEA/XTEA:
0x9E3779B9 (golden ratio)
By structure:
- Block cipher: Fixed-size blocks, padding
- Stream cipher: Byte-by-byte, no padding
- Hash: Mixing functions, rounds, constants
Telegram Bot API for Evidence Recovery
Pattern (Stomaker): Malware uses Telegram bot to exfiltrate stolen data.
Recover exfiltrated data via bot token:
import requests
TOKEN = "bot_token_here"
r = requests.get(f"https://api.telegram.org/bot{TOKEN}/getUpdates")
file_id = "..."
r = requests.get(f"https://api.telegram.org/bot{TOKEN}/getFile?file_id={file_id}")
file_path = r.json()['result']['file_path']
requests.get(f"https://api.telegram.org/file/bot{TOKEN}/{file_path}")
Poison Ivy RAT Traffic Decryption (Trend Micro CTF 2015)
Pattern: PCAP contains Poison Ivy RAT (Remote Access Trojan) traffic. Poison Ivy uses Camellia cipher with the key derived from an attacker-supplied password (null-padded to key length). The default password is "admin".
chopshop -f capture.pcap -s ./output/ "poisonivy_23x -c -w admin"
Identification:
- Traffic to non-standard ports (often 3460, 65535)
- Initial handshake with 256-byte key exchange
- Encrypted data blocks with 8-byte aligned lengths
Alternative decryption (Python):
from Crypto.Cipher import Camellia
password = b"admin"
key = password.ljust(32, b'\x00')[:32]
cipher = Camellia.new(key, Camellia.MODE_ECB)
plaintext = cipher.decrypt(encrypted_data)
Key insight: Poison Ivy's encryption key is derived solely from the attacker password with null-byte padding — no key derivation function. The default password "admin" is commonly left unchanged. ChopShop with poisonivy_23x module automates full session reconstruction (screenshots, file listings, keystrokes). Also try common passwords: "password", "p0ison", or challenge-provided hints.
PE, .NET, and Binary Malware Analysis
Table of Contents
PE Analysis
peframe malware.exe
pe-sieve
pestudio
Sandbox Evasion Checks
Look for:
- VM detection (VMware, VirtualBox artifacts)
- Debugger detection (IsDebuggerPresent)
- Timing checks (sleep acceleration)
- Environment checks (username, computername)
- File/registry checks for analysis tools
Malware Configuration Extraction
Common storage locations:
- .data section (hardcoded)
- Resources (PE resources, .NET resources)
- Registry keys written at install
- Encrypted config file dropped to disk
Extraction tools:
wrestool -x -t 10 malware.exe -o config.bin
monodis --mresources malware.exe
objdump -s -j .rdata malware.exe
.NET DNS-based C2
Pattern: Deobfuscated .NET malware with DNS C2
Analysis with dnSpy:
- Find network functions (TcpClient, DnsClient, etc.)
- Identify encoding/encryption wrappers
- Look for command dispatch (switch on opcode)
AsmResolver for programmatic analysis:
using AsmResolver.DotNet;
var module = ModuleDefinition.FromFile("malware.dll");
foreach (var type in module.GetAllTypes()) {
foreach (var method in type.Methods) {
}
}
.NET Malware Analysis (C2 Extraction)
Tools: ILSpy, dnSpy, dotPeek
LimeRAT C2 extraction (Whisper Of The Pain):
- Open .NET binary in dnSpy
- Find configuration class with Base64 encoded string
- Identify decryption method (typically AES-256-ECB with derived key)
- Key derivation: MD5 of hardcoded string -> first 15 + full 16 bytes + null = 32-byte key
- Decrypt: Base64 decode -> AES-ECB decrypt -> reveals C2 IP/domain
from Crypto.Cipher import AES
import hashlib, base64
key_source = '${8\',`d0}n,~@J;oZ"9a'
md5 = hashlib.md5(key_source.encode()).hexdigest()
key = bytes.fromhex(md5[:30] + md5 + '00')[:32]
cipher = AES.new(key, AES.MODE_ECB)
plaintext = cipher.decrypt(base64.b64decode(encrypted_b64))
PyInstaller + PyArmor Unpacking
python pyinstxtractor.py malware.exe
ctf-malware — Quick Reference
Inline code / one-liners / common payloads. Loaded on demand from SKILL.md. Detailed techniques live in the category-specific support files listed in SKILL.md.
Obfuscated Scripts
- Replace
eval/bash with echo to print underlying code; extract base64/hex blobs and analyze with file. See scripts-and-obfuscation.md.
JavaScript & PowerShell Deobfuscation
- JS: Replace
eval with console.log, decode unescape(), atob(), String.fromCharCode().
- PowerShell: Decode
-enc base64, replace IEX with output. See scripts-and-obfuscation.md.
Junk Code Detection
- NOP sleds, push/pop pairs, dead writes, unconditional jumps to next instruction. Filter to extract real
call targets. See scripts-and-obfuscation.md.
PCAP & Network Analysis
tshark -r file.pcap -Y "tcp.stream eq X" -T fields -e tcp.payload
Look for C2 on unusual ports. Extract IPs/domains with strings | grep. See c2-and-protocols.md.
Custom Crypto Protocols
- Stream ciphers share keystream state for both directions; concatenate ALL payloads chronologically.
- ChaCha20 keystream extraction: send nullbytes (0 XOR anything = anything). See c2-and-protocols.md.
C2 Traffic Patterns
- Beaconing, DGA, DNS tunneling, HTTP(S) with custom headers, encoded payloads. See c2-and-protocols.md.
RC4-Encrypted WebSocket C2
- Remap port with
tcprewrite, add RSA key for TLS decryption, find RC4 key in binary. See c2-and-protocols.md.
Identifying Encryption Algorithms
- AES:
0x637c777b S-box; ChaCha20: expand 32-byte k; TEA/XTEA: 0x9E3779B9; RC4: sequential S-box init. See c2-and-protocols.md.
AES-CBC in Malware
- Key = MD5/SHA256 of hardcoded string; IV = first 16 bytes of ciphertext. See c2-and-protocols.md.
PE Analysis
peframe malware.exe
pe-sieve
pestudio
See pe-and-dotnet.md.
.NET Malware Analysis
- Use dnSpy/ILSpy for decompilation; AsmResolver for programmatic analysis. LimeRAT C2: AES-256-ECB with MD5-derived key. See pe-and-dotnet.md.
Malware Configuration Extraction
- Check .data section, PE/.NET resources, registry keys, encrypted config files. See pe-and-dotnet.md.
Sandbox Evasion Checks
- VM detection, debugger detection, timing checks, environment checks, analysis tool detection. See pe-and-dotnet.md.
Anti-Analysis Techniques
VM detection (CPUID, MAC prefix, registry, disk size), timing evasion (sleep/RDTSC sandbox detection), API hashing (ROR13/DJB2/CRC32 + hashdb lookup), process injection (hollowing, APC, CreateRemoteThread), environment checks. See scripts-and-obfuscation.md.
PyInstaller + PyArmor Unpacking
pyinstxtractor.py to extract, PyArmor-Unpacker for protected code. See pe-and-dotnet.md.
Telegram Bot Evidence Recovery
- Use bot token from malware source to call
getUpdates and getFile APIs. See c2-and-protocols.md.
Debian Package Analysis
ar -x package.deb && tar -xf control.tar.xz
See scripts-and-obfuscation.md.
YARA Rules for Malware Detection
Write YARA rules to match byte patterns, strings, and regex against files or memory dumps. Detect XOR loops ({31 ?? 80 ?? ?? 4? 75}), base64 blobs, encoded PowerShell. Use yarac to compile for faster scanning. See scripts-and-obfuscation.md.
Shellcode Analysis
Disassemble with objdump -b binary -m i386:x86-64, emulate with Unicorn Engine (hook syscalls safely), or use Capstone for programmatic disassembly. Look for XOR decoder stubs. See scripts-and-obfuscation.md.
Memory Forensics for Malware
vol3 windows.malfind detects injected code (PAGE_EXECUTE_READWRITE without mapped file). windows.pstree reveals suspicious parent-child relationships. YARA scan memory with yarascan.YaraScan. See scripts-and-obfuscation.md.
Network Indicators Quick Reference
strings malware | grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
tshark -r capture.pcap -Y "dns.qry.name" -T fields -e dns.qry.name | sort -u
Scripts and Obfuscation Analysis
Table of Contents
Obfuscated Scripts (General)
- Replace
eval/bash with echo to print underlying code
- Extract base64/hex blobs and analyze with
file
- Common deobfuscation chain: base64 decode -> gzip decode -> reverse -> base64 decode
JavaScript Deobfuscation
eval = console.log;
unescape()
String.fromCharCode()
atob()
PowerShell Analysis
# Common obfuscation
-enc / -EncodedCommand # Base64 encoded
IEX / Invoke-Expression # Eval equivalent
[System.Text.Encoding]::Unicode.GetString([System.Convert]::FromBase64String($encoded))
Junk Code Detection
Pattern: Obfuscation adds meaningless instructions around real code
Identification:
- NOP sleds, push/pop pairs that cancel
- Arithmetic that results in zero/identity
- Dead writes (register written but never read before next write)
- Unconditional jumps to next instruction
Filtering technique:
def extract_real_calls(disassembly):
calls = []
for instr in disassembly:
if instr.mnemonic == 'call' and not is_junk_target(instr.operand):
calls.append(instr)
return calls
Hex-Encoded Payloads
- Convert hex to bytes, try common transformations: subtract 1, XOR with key
Debian Package Analysis
ar -x package.deb
tar -xf control.tar.xz
Dynamic Analysis Techniques
strace -f -e trace=network,file -o trace.log ./malware
ltrace -f -o ltrace.log ./malware
sudo tcpdump -i any -w malware_traffic.pcap &
sudo tcpdump -i any port 53 -l | tee dns_queries.log &
timeout 60 ./malware
inotifywait -m -r /tmp /var/tmp --format '%T %w%f %e' --timefmt '%H:%M:%S' &
./malware
watch -n 1 'ps aux | grep -v grep | grep malware'
pid=$(pgrep malware)
strings /proc/$pid/maps
cat /proc/$pid/mem 2>/dev/null | strings | grep -i flag
import subprocess, os, tempfile
def run_sample(path, timeout=30):
"""Run malware sample with monitoring"""
with tempfile.NamedTemporaryFile(suffix='.pcap', delete=False) as pcap:
tcpdump = subprocess.Popen(
['sudo', 'tcpdump', '-i', 'any', '-w', pcap.name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
result = subprocess.run(
['strace', '-f', '-e', 'trace=network,file', path],
capture_output=True, text=True, timeout=timeout)
print("STDOUT:", result.stdout[:500])
print("STDERR (syscalls):", result.stderr[:2000])
except subprocess.TimeoutExpired:
print(f"Sample ran for {timeout}s (killed)")
finally:
tcpdump.terminate()
print(f"PCAP saved: {pcap.name}")
Key insight: Dynamic analysis reveals runtime behavior that static analysis misses: actual C2 domains resolved, encryption keys in memory, dropped files, and anti-analysis checks that were bypassed. Always run in an isolated environment (VM snapshot, Docker container) and monitor network, filesystem, and process activity simultaneously.
YARA Rules for Malware Detection
cat > detect_malware.yar << 'EOF'
rule SuspiciousStrings {
meta:
description = "Detect common malware indicators"
strings:
$s1 = "cmd.exe /c" nocase
$s2 = "powershell -enc" nocase
$s3 = {4D 5A 90 00} // MZ header (hex pattern)
$s4 = /https?:\/\/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/ // IP-based URL
$xor_loop = {31 ?? 80 ?? ?? 4? 75} // XOR decode loop pattern
condition:
2 of ($s*) or $xor_loop
}
EOF
yara detect_malware.yar suspicious_file.exe
yara -r detect_malware.yar /path/to/directory/
yara detect_malware.yar memory.dmp
Common YARA patterns for CTFs:
rule Base64_PowerShell {
strings:
$enc = "powershell" nocase
$b64 = /[A-Za-z0-9+\/]{50,}={0,2}/
condition:
$enc and $b64
}
rule XOR_Encrypted_PE {
strings:
$mz = {4D 5A}
condition:
not $mz at 0 and filesize < 1MB
// PE without MZ header = likely XOR encrypted
}
Key insight: YARA rules match byte patterns, strings, and regex against files or memory. In CTFs, write rules to detect specific obfuscation patterns (XOR loops, base64 blobs, encoded PowerShell), then apply to memory dumps or malware samples. Use yarac to compile rules for faster scanning.
Shellcode Analysis
objdump -d shellcode.bin -b binary -m i386:x86-64 -M intel
python3 << 'PYEOF'
from unicorn import *
from unicorn.x86_const import *
shellcode = open('shellcode.bin', 'rb').read()
mu = Uc(UC_ARCH_X86, UC_MODE_64)
BASE = 0x400000
STACK = 0x7fff0000
mu.mem_map(BASE, 0x1000)
mu.mem_map(STACK - 0x1000, 0x2000)
mu.mem_write(BASE, shellcode)
mu.reg_write(UC_X86_REG_RSP, STACK)
def hook_syscall(mu, user_data):
rax = mu.reg_read(UC_X86_REG_RAX)
print(f"syscall: {rax}")
mu.hook_add(UC_HOOK_INSN, hook_syscall, None, 1, 0, UC_X86_INS_SYSCALL)
mu.emu_start(BASE, BASE + len(shellcode))
PYEOF
python3 -c "
from capstone import *
md = Cs(CS_ARCH_X86, CS_MODE_64)
code = open('shellcode.bin','rb').read()
for i in md.disasm(code, 0x0):
print(f'{i.address:#x}: {i.mnemonic} {i.op_str}')
"
scdbg /f shellcode.bin
Key insight: Shellcode in CTF malware challenges is often XOR-encoded or staged. Look for decoder stubs (short loops with XOR), then extract and decode the payload. Unicorn Engine emulation is safer than running shellcode — it intercepts syscalls without executing them.
Memory Forensics for Malware
vol3 -f memory.dmp windows.pslist
vol3 -f memory.dmp windows.pstree
vol3 -f memory.dmp windows.psscan
vol3 -f memory.dmp windows.memmap --pid PID --dump
vol3 -f memory.dmp windows.malfind
vol3 -f memory.dmp windows.netscan
vol3 -f memory.dmp windows.cmdline
vol3 -f memory.dmp windows.dlllist --pid PID
vol3 -f memory.dmp yarascan.YaraScan --yara-rules "rule test { strings: $s = \"flag{\" condition: $s }"
Key insight: windows.malfind detects injected code by finding memory regions with PAGE_EXECUTE_READWRITE protection and no corresponding mapped file — the hallmark of process injection. Combine with windows.pstree to find processes with unexpected parent-child relationships (e.g., svchost.exe spawned by cmd.exe).
Anti-Analysis Techniques
Malware uses runtime checks to detect analysis environments and alter behavior. Bypass these to reach the actual malicious functionality.
VM / Sandbox Detection
Pattern: Malware checks for virtualization artifacts before executing payload. In CTFs, the "real" flag logic is behind these checks.
Key insight: Identify the detection method, then patch the check or fake the environment.
VM_ARTIFACTS = [
'vmtoolsd.exe', 'vmwaretray.exe', 'VBoxService.exe',
'qemu-ga.exe', 'sandboxie', 'wireshark.exe',
'/sys/class/dmi/id/product_name',
'C:\\windows\\system32\\drivers\\vmmouse.sys',
]
Timing-Based Evasion
Pattern: Malware uses sleep(), GetTickCount(), or RDTSC to detect accelerated execution in sandboxes.
Key insight: Look for calls to sleep, time.sleep, NtDelayExecution, GetTickCount64, QueryPerformanceCounter. If the sample just sits there doing nothing, it's likely in a sleep-based anti-sandbox check.
API Hashing
Pattern: Instead of importing functions by name (visible in strings/imports), malware resolves API addresses at runtime by hashing function names and comparing to hardcoded hash values.
def ror13_hash(name):
h = 0
for c in name:
h = ((h >> 13) | (h << 19)) & 0xFFFFFFFF
h = (h + ord(c)) & 0xFFFFFFFF
return h
def djb2_hash(name):
h = 5381
for c in name:
h = ((h * 33) + ord(c)) & 0xFFFFFFFF
return h
import binascii
def crc32_hash(name):
return binascii.crc32(name.encode()) & 0xFFFFFFFF
Key insight: When strings output shows almost no readable API names but the binary clearly does complex operations, suspect API hashing. Look for the hash function (small loop with XOR/rotate/add), then use hashdb or build a rainbow table against kernel32.dll and ntdll.dll exports.
Process Injection Techniques
Pattern: Malware injects code into legitimate processes to evade detection. Understanding the injection method helps extract the actual payload.
vol3 -f memory.dmp windows.malfind
vol3 -f memory.dmp windows.hollowfind
vol3 -f memory.dmp windows.malfind --dump --pid <PID>
Environment Variable / Hostname Checks
Pattern: Malware checks for specific environment conditions (hostname, username, domain, locale) to target specific victims or avoid analysis labs.
Key insight: If a malware sample exits immediately or behaves differently than expected, trace its API calls with strace/ltrace or step through with a debugger. Look for string comparisons against environment values early in execution.
VSCode .vsix onStartupFinished Activation → Obfuscated child_process Exfil (source: HTB University 2025 Snowy Extension)
Trigger: forensics artefact: a .vsix file (VSCode extension) with package.json listing "activationEvents": ["onStartupFinished"]; extension.js contains base64-heavy globals + child_process.
Signals: unzip .vsix (it's a ZIP) → extension.js > 50 KB with high-entropy string blobs; onStartupFinished activation event.
Mechanic: extension auto-runs at VSCode launch without user interaction (differs from classic npm supply-chain). Reverse steps:
unzip x.vsix -d ext/
- deobfuscate
extension.js with webcrack or hand-rolled AST walker (usually JSFuck / obfuscator.io layers)
- identify exfil endpoint, persistence via
context.globalState.update('key', ...), and any shelled-out child_process.spawn calls
- craft YARA for the activation-event + child_process pattern
rule vsix_startup_child_process {
strings:
$a = "onStartupFinished"
$b = "child_process"
$c = "globalState.update"
condition:
uint32(0) == 0x04034b50 and 2 of them
}