Post

Reverse Engineering technique repository (map)

Reverse Engineering technique repository (map)

Every technique in the series, grouped by topic, with when to use it and the matching lesson. In the lesson notation, 15.5 = Part 15, Lesson 5 (see the README).

A. Static analysis

Don’t run the file, just read.

TechniqueUse whenLesson
Triage / fingerprintingAlways the first step: know the file type, compiler, packer2.1, 14.1
Reading strings & stacked/encoded stringsQuick clues: URLs, paths, error messages, flags2.1
Import/export analysisInfer functionality from the APIs called1.7, 1.13
Reading disassemblyUnderstand each machine instruction1.3,1.5
Reading decompiler outputQuick understanding at the C/pseudocode level2.2, 2.3
Recovering data types & structsMake pseudocode readable3.3, 4.2
Cross-reference (xref)Trace where a function/variable/string is used2.2
Identifying the calling conventionRead parameters correctly1.4
Identifying control structures (if/loop/switch)Translate assembly back to logic1.5
FLIRT / library signaturesSkip library code, focus on the author’s code3.4
Control Flow Graph (CFG)See the overall flow of a function2.2, 2.3

B. Dynamic analysis

Run the file in a controlled environment.

TechniqueUse whenLesson
Debugging (breakpoint, step, watch)Observe real values at runtime2.5, 2.6
API breakpointsStop at GetProcAddress, CreateFile…2.5, 15.9
Memory breakpoint / hardware breakpointCatch accesses to a memory region2.5
Memory dumpGet code that has been decrypted/unpacked14.2, 14.3
Tracing (API/syscall/library)Understand overall behavior2.8, 17.2
System monitoring (Procmon…)See file/registry/network effects2.8, 19.1
Time Travel DebuggingRewind to find where a value came from2.6
Network monitoringUnderstand the C2/API protocol2.8, 18.7

C. By language / platform

PlatformSpecific techniquesLesson
C/C++ nativeRecovering structs, vtables, RTTI, name mangling3.x, 4.x
.NETDecompile IL to C#, debug without source, patch IL5.x
Java/AndroidDEX to Java, smali patching, re-signing APKs, hooking Java6.x
Python.pyc to source, unpacking PyInstaller7.x
GoRecovering symbols from pclntab8.x
RustDemangling, recognizing Result/Option patterns9.x
JS/Electron/WASMDeobfuscating, unpacking asar, wasm2wat11.x
AppleObjC runtime, Swift demangling12.x
GamesIL2CPP dump, Cheat Engine, Lua decompiling13.x

D. Unpacking & deobfuscation

TechniqueDescriptionLesson
Identifying packers & entropyTell packed from unpacked14.1
Automatic unpacking (UPX -d, unipacker)Fast when the packer is standard14.2
Manual unpacking (finding the OEP)For custom packers; follow the tail jump/ESP trick14.2
Dump + rebuild IAT (Scylla)Recreate a runnable file after unpacking14.3
Removing string encryptionDecrypt statically encrypted strings14.4, 18.2
Removing control-flow flatteningRecover the original flow14.4, 14.6
Removing opaque predicates / MBASimplify junk expressions14.4, 18.3
Dealing with virtualization (VMProtect/Themida)Understand the bytecode handlers14.5
Deobfuscating with emulation/symbolic executionAutomation14.6, 18.2, 18.3

E. Anti-reverse (recognizing & getting past)

These are covered so you understand the mechanism, to analyze and defend.

Software-side techniqueHow the reverser handles itLesson
Anti-debug via APIs (IsDebuggerPresent…)Hook/patch to return fake values, ScyllaHide15.1, 15.9
Anti-debug via PEB/NtGlobalFlagEdit the flags in memory15.2
Timing anti-debug (RDTSC)Skip/adjust the delta, patch15.3
Traps (INT3/INT2D/ICEBP), hardware BP detectionRecognize and avoid15.3
Self-debug, debug object, thread hidingHyperHide/TitanHide15.4, 15.9
TLS callbacks running before mainSet a BP at the TLS callback15.4
Anti-VM/sandboxMake the VM look real, patch the checks15.5
Anti-disassembly (junk/overlap/SMC)Fix the format, run dynamically, force code15.6
Anti-attach / anti-dump / anti-hookAttach early, rebuild headers, compare prologues15.7
Integrity check (CRC/checksum)Disable the check instead of editing the checked code15.8

F. Crypto & algorithms

TechniqueDescriptionLesson
Identifying crypto constantsfindcrypt/capa/signsrch16.1
Identifying XOR/RC4/custom Base64The most common patterns in malware/crackmes16.2
Identifying AES/DES/TEA/ChaCha/hashesThrough S-boxes, constants, round structure16.3
Rewriting the algorithm in PythonTo solve it yourself/write a keygen16.4
Solving conditions with Z3/angrWhen the check logic is complex16.4, 18.3

G. Patching, hooking, injection, instrumentation

TechniqueDescriptionLesson
Static patching (change jumps, NOP, code cave)Permanently change behavior in the file17.1
Runtime patchingEdit in memory while running17.1
IAT hookReplace a pointer in the import table17.3
Inline/trampoline hook (Detours/MinHook)Insert a jump at the function start17.3
Frida Interceptor/StalkerFlexible hooking & tracing on any platform17.2
DLL injection (LoadLibrary+CreateRemoteThread, SetWindowsHookEx, AppInit)Load code into another process17.4
Manual mapping / reflective DLLLoad without going through the standard loader17.4
Shellcode injection, APC, thread hijacking, process hollowingMechanisms & detection signs (PE-sieve, EDR)17.5
LD_PRELOAD / ptrace / DYLD_INSERT_LIBRARIESEquivalents on Linux/macOS17.6
DBI (Pin/DynamoRIO/QBDI/TinyInst)Taint, coverage, large-scale tracing17.7

H. Advanced & automation

TechniqueDescriptionLesson
Decompiler scripting (IDAPython/Ghidra/BN API)Automate repetitive analysis18.1
Emulation (Unicorn/Qiling/Speakeasy)Run an isolated piece of code18.2
Symbolic execution (angr/Triton)Automatically find inputs satisfying conditions18.3
Binary/patch diffingFind vulnerabilities from patches, compare variants18.4
Firmware/IoT (binwalk/QEMU)Reverse embedded devices18.5
Kernel driver / LKMReverse ring-0 code18.6
Reversing protocols/file formatsRebuild the spec from samples18.7
AI assistance (LLM plugin/MCP)Speed up naming and explanation18.8

I. Malware analysis (defensive)

TechniqueDescriptionLesson
Safe workflow + sandboxRun samples without spreading infection19.1
Writing IOC/YARA/capa/sigmaDetection & threat hunting19.2
Maldoc analysis (macro/PDF/LNK)The initial infection stage19.3
Extracting config & C2Understand the attacker’s infrastructure19.4

Suggested skill path

Start with basic static analysis (A) until you can read disassembly and pseudocode, then move to basic dynamic analysis (B) and get comfortable with x64dbg/GDB. Pick one managed language (.NET or Java) to see results quickly, then take on native C/C++. Unpacking and anti-reverse (D, E) come next, and they’re the hard part for most people. After that comes automation (H) with scripting, emulation and symbolic execution. Finally, specialize in one direction, either malware, exploit/vuln research, or games/mobile.

This post is licensed under CC BY 4.0 by the author.