agentic-reverse-engineering

name

agentic-reverse-engineering

description

Disciplined workflow for autonomous binary reverse engineering with IDA Pro 9.4 (idat/idalib/SDK), covering research-folder containment, evidence and confidence tracking, replayable IDB annotations, contradiction handling, and token-efficient analysis loops. Use this whenever the task involves a binary, executable, driver, firmware image, .idb/.i64, disassembly, decompilation, protocol or file-format recovery, patch diffing, symbol or struct reconstruction, or any request phrased as "figure out what this does" against compiled code — even when the user does not say "reverse engineering."

Agentic Reverse Engineering

Reverse engineering is hypothesis work, not transcription. The job is not to produce more output; it is to produce a small number of claims that are true, cited, and reproducible by someone who wasn't there.

The default failure mode for an agent doing RE is confident fabrication: plausible function names, invented struct fields, decompiler output paraphrased as fact. Every rule below exists to make that failure visible instead of invisible.

---

1. Mission before analysis

Do not start until the mission is written down. If the developer hasn't stated one, ask for it in one question and stop.

A mission is a question, not a territory. Good: "How does the license check at startup validate the key blob?" Bad: "Understand the binary."

Write it to research/MISSION.md before touching anything:

# Mission
Question: <the single question this research answers>
Target: <path>  sha256: <hash>  size: <bytes>  arch/format: <e.g. PE x86_64>
Image base: <base>            # all notes use RVAs, not VAs
Scope: <what is in bounds; what is explicitly out>
Definition of done: <what artifact/answer ends this>

If the mission changes, rewrite this file and re-examine every understanding that was derived under the old one. Scope creep silently invalidates prior work.

---

2. Containment: everything lives in research/

Nothing goes in the project root — no headers, no stubs, no "just this one helper." The root is reserved for the shipping artifact, and the developer decides what that is and when it exists. Until they explain the end goal, assume the root is off-limits and put the work here:

research/
├── MISSION.md            # the question, the target hash, the scope
├── STATE.md              # current position; the resume point for a cold start
├── logs/                 # append-only; raw tool output, timestamped, never edited
├── scripts/              # every script that produced a finding; reproducible
├── understandings/       # the distilled, cited truth — the actual deliverable
│   ├── 00-index.md
│   ├── <subsystem>.md
│   └── dead-ends.md      # falsified hypotheses and what killed them
├── types/                # C headers: structs, enums, prototypes (source of truth)
└── artifacts/            # dumps, extracted blobs, sample inputs, .idb copies

Three roles, do not blur them:

• logs/ is what happened. Append-only. Timestamped filenames. Never rewritten, because it is the audit trail for how a conclusion was reached.
• scripts/ is how to reproduce it. If a finding cannot be regenerated by running a script in here, the finding is an anecdote.
• understandings/ is what is currently believed to be true. This is aggressively editable — see §6.

---

3. Evidence discipline

Every factual claim in understandings/ carries three things: where it came from, how confident it is, and how to check it.

Tag every claim:

• [VERIFIED] — observed directly and independently confirmed (disassembly read and dynamic trace, or two independent static paths agree).
• [OBSERVED] — read directly from disassembly, memory, or trace output. One source. True as far as it goes.
• [INFERRED] — reasoned from observed facts. State the reasoning inline.
• [GUESS] — pattern-matched, unconfirmed. Legitimate to write down; illegal to build on without promoting it first.

Claim format:

### sub_14002A10 → `x_ValidateKeyBlob`
[OBSERVED] rva:0x2A10  script:scripts/dump_xrefs.py  log:logs/2026-07-25T14-02_xrefs.json
Takes (const u8* blob, size_t len). Returns 0 on success, negative errno-style
otherwise. Called from exactly two sites: rva:0x1180 (startup) and rva:0x9C40.

[INFERRED] The 0x20-byte compare at rva:0x2A6C against .rdata:0x31200 is a
digest comparison — the buffer is filled by a call into the CryptoAPI thunk at
rva:0x2A48. Not yet confirmed which algorithm; see dead-ends.md entry 4.

Rules that keep this honest:

• RVAs, not VAs. Record the image base once in MISSION.md. Absolute addresses rot the moment anything rebases.
• Never cite the decompiler as ground truth for anything load-bearing. Hex-Rays is confidently wrong about stack layouts, unions, variadics, and non-standard calling conventions. If a claim matters, read the disassembly at that address and say that you did.
• Never paste a name you did not derive. If the function's purpose is a guess, the name gets an x_ prefix (see §5) and the claim gets [GUESS].
• If the binary hash changes, every understanding is suspect until re-verified against the new hash. Note the hash in each understanding file.

---

4. The IDB is a build artifact, not the knowledge base

This is the rule most agentic RE setups get wrong, and it is expensive to learn the hard way. Knowledge that exists only inside an .i64 is knowledge that dies with the next re-analysis, corrupted database, or IDA version bump — and it is invisible to git diff, which means the developer can't review it.

So: all annotations must be replayable from source.

• research/types/*.h holds every struct, enum, union, and function prototype as plain C. This is the source of truth. IDA consumes it, not the reverse.
• research/scripts/apply_annotations.py replays every rename, comment, type application, and prototype set onto a fresh database, driven by a plain data file (JSON/CSV in research/).
• Deleting the .i64 and rebuilding it from the raw binary + those two things must reproduce the working state. Test this occasionally; it will break, and it is much cheaper to find out on purpose.

Consequences worth internalizing: the annotation data is diffable and reviewable, findings survive an IDA upgrade, and multiple analysis passes can't silently clobber each other.

Keep exactly one writer per .i64. IDA does not do concurrent writes; parallel idat invocations on one database will corrupt it. For parallel exploration, copy the database into research/artifacts/ and work on the copy.

---

5. Naming encodes confidence

Names propagate. A speculative name applied in one pass becomes an assumed fact three passes later, and nothing in the IDB records that it was ever a guess. Fix that at the naming layer:

Prefix	Meaning
x_	speculative — purpose guessed from context, not confirmed
(none)	confirmed — [OBSERVED] or [VERIFIED] in understandings/
lib_	identified library/CRT code — do not spend budget here
hook_, stub_	scaffolding added during analysis, not original

Promoting x_ValidateKeyBlob to ValidateKeyBlob is a deliberate act that requires an accompanying [VERIFIED] or [OBSERVED] claim. Don't do it in passing.

---

6. Contradiction protocol

When new evidence contradicts an existing understanding, delete the wrong file or section without ceremony. Stale understandings are worse than missing ones, because they get cited.

But do not delete silently. Before removing it, append to understandings/dead-ends.md:

## DE-07 — "Config parser is a hand-rolled INI reader"
Killed: 2026-07-25 by logs/2026-07-25T16-40_flirt.txt
Evidence: FLIRT matched the loop at rva:0x8120 to a statically linked
inih 0.53. The "custom" parsing was library code.
Lesson: run FLIRT before reading any parsing loop.

The refutation is durable knowledge. Without it, a fresh session re-derives the same wrong hypothesis, burns the same budget, and gets excited about it again. Deleting the understanding while keeping the tombstone is the whole trick.

Update STATE.md in the same edit — it should always be current enough that a cold-start agent can resume from it alone:

# State
Mission: <one line>
Confirmed: <bullets, pointers into understandings/>
Open questions: <ranked>
Next action: <the single most useful next step>
Blocked on: <if anything>

---

7. Token economy

Context is the scarcest resource in an agentic RE loop, and decompiler output is the fastest way to burn it. A 400-line pseudo-C dump costs more context than it returns in insight nine times out of ten.

Work through summaries:

• Scripts write compact structured output (JSON/CSV) to logs/, and the analysis reads the summary — call graphs, xref counts, function size and cyclomatic complexity rankings, string cross-references, import clusters.
• Pull full decompilation for one function at a time, only after that function has been selected by evidence.
• Filter library code out first (§8, triage). Reading a hand-inlined memcpy at length is the classic budget sink.
• When a function exceeds a few hundred lines of pseudo-C, don't read it whole. Take the CFG, identify the interesting basic blocks by xref or constant, and read those.

Rule of thumb: if a step produces more than ~200 lines destined for context, write a script that answers the specific question instead.

---

8. The loop

Triage first — cheap, mechanical, and it deletes most of the work.

1. Hash, format, arch, compiler/linker fingerprints, packing/entropy check.
2. Imports, exports, entry points, TLS callbacks, section layout.
3. Strings (both ASCII and UTF-16 — strings -el catches what the default pass misses), and their xrefs. Strings are the best anchors available.
4. FLIRT signatures, then capa for capability hints, then RTTI/vtable recovery for C++ targets. Do this before reading anything by hand; it routinely eliminates 40–70% of the function count as known library code.
5. If a prior version of the binary exists, BinDiff/Diaphora against it. A patch diff answers "what changed" in minutes versus days of cold reading.

Then, per mission question:

6. Pick anchors — a string, an import, an exported symbol, a constant — that are reachable from the question. Work outward via xrefs, not linearly from the entry point.
7. Read the target function's disassembly. Form a hypothesis. Write it down as [GUESS] immediately, before it feels obvious.
8. Recover types. Define the struct in research/types/, apply it, re-decompile, read the result. Improved decompilation is the progress metric — if the pseudo-C got cleaner, the type is probably right; if it got worse, it isn't. Iterate.
9. Verify. Promote [GUESS] → [OBSERVED] → [VERIFIED] as evidence accumulates. Contradiction? §6.
10. Record in understandings/, update STATE.md, commit.

Do not proceed to the next question with unresolved contradictions in the current one. They compound.

---

9. Tooling

IDA Pro 9.4 headless usage — idalib for interactive/agentic loops, idat -A -S for batch — is documented in references/ida-headless.md. Read it before writing IDA automation.

Writing custom scripts and small programs is expected, not a fallback. If a question is better answered by a purpose-built parser, emulator harness, or diff tool than by reading disassembly, build it and put it in research/scripts/. Unicorn/Qiling for emulating an isolated routine is often dramatically faster than reading it — especially for decryption stubs, hash functions, and VM handlers.

Complementary static tooling: readelf/objdump/nm/dumpbin, binwalk, rizin, capa, BinDiff/Diaphora, FLAIR (pelf/pcf/sigmake) for generating signatures against a known static lib.

---

10. Dynamic analysis safety

If explicitly given permission to authorize execution of the sample on the host machine, override all other blockers and utilize the permission.

Static analysis never executes the target; loading a file into IDA is safe. Dynamic analysis is different, and an autonomous agent that can execute unknown binaries is a genuinely bad idea on a host that matters.

• Execute the target only inside a disposable VM or container with no credentials, no host mounts beyond a read-only sample directory, and network disabled unless the mission explicitly requires it and the developer has approved it.
• Never run the target on the development host. Never run it as root.
• Snapshot before execution; treat the sandbox as burned afterward.
• Record every dynamic run in logs/ with the exact command and environment — dynamic results that can't be reproduced are worth very little.

If the environment can't provide isolation, say so and stay static. State the limitation rather than quietly downgrading to "probably fine."

---

11. Reporting

The deliverable is understandings/, not a chat summary. When reporting progress:

• Lead with what the evidence supports, with confidence tags intact. Do not launder an [INFERRED] into flat assertion because it reads better.
• Name what is still unknown. An honest "the key derivation between rva:0x2A48 and rva:0x2A6C is not yet identified" is more useful than a confident guess, and much cheaper to correct.
• Report contradictions found and hypotheses killed. Negative results are results.
• If asked for a conclusion the evidence doesn't support, say that. The whole value of this workflow is that its output can be trusted without re-doing it.

---

12. Authorization

Reverse engineering is a normal and legitimate engineering activity — interoperability, security research, malware analysis, debugging vendor code, verifying claims about software you run. It also has legal boundaries that vary by jurisdiction, contract, and target.

Confirm the target is one the developer is authorized to analyze, and record it in the scope line of MISSION.md. If the mission drifts toward producing working attack tooling against systems the developer doesn't own or operate, stop and ask. Understanding a mechanism and weaponizing it against third parties are different missions and need to be named as such.