CTF reverse engineering methodology
1. Triage
file ./bin
strings ./bin | head -50
strings ./bin | grep -i "flag\|ctf{\|password\|key"
xxd ./bin | head -20 # magic bytes
checksec --file=./bin # protections
2. Detect packing
# Entropy check (>7.5 = packed/encrypted)
ent ./bin # or: python3 -c "from collections import Counter; ..."
# UPX / known packers
upx -t ./bin # tests + identifies UPX
detect-it-easy-cli ./bin
diec ./bin
If UPX-packed: upx -d ./bin -o unpacked. For custom packers: dump from memory after unpacking stub runs (gdb / x64dbg).
3. Pick the decompiler
| Binary type | Best tool |
|---|---|
| ELF / PE / Mach-O | Ghidra (registry), IDA (commercial), Binary Ninja |
| Stripped ELF | Ghidra + recover symbols via FunctionID / Lumen |
| .NET (DLL/EXE) | dnSpyEx, ilspycmd, dotPeek |
| Java JAR | jadx, cfr, procyon |
| Java class | javap -c -p |
| Android APK | jadx-gui, apktool d then jadx on dex |
| iOS / Mach-O | Hopper, Ghidra |
| Go binary | redress, GoReSym, Ghidra + Go plugin |
| Rust | Ghidra + rustfilt for symbols |
| WASM | wabt (wasm-decompile), wasmer for run |
Python .pyc |
uncompyle6, decompyle3, pycdc |
| PyInstaller .exe | pyinstxtractor then pycdc on .pyc |
| Compiled Lua | unluac, luadec |
4. Dynamic analysis
# Trace
ltrace ./bin
strace ./bin
strace -f -e trace=read,write,open ./bin
# Debugger
gdb-multiarch ./bin
# pwndbg or gef extensions are loaded by default
# Fault injection / branch flipping
gdb> set $eax = 1 # change return value to bypass check
For Android: frida for runtime instrumentation, objection on top.
5. Anti-debug / anti-VM
Common checks:
ptrace(PTRACE_TRACEME)returns -1 if already debugged → patch withnop/proc/self/statusTracerPid: 0check → LD_PRELOAD a fake or patch- timing checks (rdtsc) → patch
IsDebuggerPresent()(Windows) → patch the call site
Tooling: ScyllaHide (Windows), gdb scripts to auto-bypass, frida to hook.
6. Custom VMs
If you see a big switch dispatcher reading a "bytecode" buffer:
- Identify the opcode handler table
- Map opcodes → operations
- Disassemble the bytecode manually (write a small Python disasm)
- Reverse the program logic in the decoded bytecode
This is common in harder rev challenges. Don't fight the VM — disassemble it.
7. Symbolic execution / SMT
For challenges where input → boolean check, and check is complex but pure:
angr— Python symbolic execution. Find path to "win" basic block.manticore— alternative.z3directly — when constraints are explicit (e.g., "input[i] ^ key[i] == ...").
Pwntools venv has z3-solver already.
8. Don't waste time on
- Reading every disassembly line top-to-bottom — find
main, focus there - Reimplementing
strings,objdump,readelf— use the tools - Manual hex editing for big patches — use
radare2or Ghidra's patch instructions
After solve
Use the writeup-template skill. Include the key reverse-engineered algorithm or VM disassembly.