CTF reverse engineering speed-run
When it applies
The handout is a binary that checks something — a flag, key, serial, or password — and you need
the accepted input. This skill is the CTF workflow: find the check fast, defang whatever stops
you from seeing it, and solve the constraints mechanically. For deep tool use defer to
reverse-eng-binary-triage (first-pass recon), reverse-eng-radare2 / reverse-eng-ghidra /
reverse-eng-ida (analysis), reverse-eng-deobfuscation (unpacking/devirtualization).
Why it works
A flag-checker is a pure function from input to accept/reject, and CTF ones are small. You never
need to understand the whole binary — only the check — so the game is locating it (strings and
xrefs beat linear reading), exposing it (anti-debug and packing are thin in CTF), and solving
it (the constraints are almost always z3-shaped; manual algebra is the slow path).
Method
- Two-minute triage.
file, strings -n 6 (+ strings -el for UTF-16), checksec, run it
with junk input. The strings that matter: "Correct!"/"Wrong", the flag format itself
(sometimes just… present), odd alphabets, and packer markers (UPX!). Packed → upx -d first;
anything fancier → reverse-eng-deobfuscation. .NET/Java/Go/Rust/Python-exe binaries route to
their dedicated tooling (reverse-eng-dotnet, reverse-eng-go-rust, pyinstxtractor+uncompyle).
- Locate the check, don't read the binary. In r2/Ghidra/IDA: find the "Wrong" string, follow
the xref to its function, and read backwards from the branch that prints it. Everything
between the input read and that branch is the check. For C/C++ this is usually one function.
- Dynamic shortcuts before static grinding.
ltrace ./chall — library calls like strcmp(input, "..."), strlen, per-char compares
leak the answer or its length with zero analysis.
- Compare-per-character binaries → count crashes/coverage: flip one input byte at a time and
watch which comparison index changes, or use a debugger on the compare loop to read each
expected byte as it's checked.
strace for flag files, env vars, network the binary secretly consults.
- Defang anti-debug (it's shallow in CTF). Common tricks and one-line counters:
ptrace(PTRACE_TRACEME) → patch the call, or catch syscall ptrace in gdb and force
return 0.
IsDebuggerPresent / PEB.BeingDebugged → zero the flag in the debugger, or patch the
conditional jump that follows.
- Timing checks (
rdtsc) → patch or ignore — they only guard a branch.
- Self-checksum / anti-tamper → make all your patches before the checksum computes, or patch
the comparison result, not the data.
The universal move: find the check that branches on the anti-debug result and flip that one
branch — don't fight each mechanism.
- Solve the check with z3 — the CTF superpower. When the check transforms input bytes and
compares against constants (xor/add/rotate/substitute chains, matrix math, per-position
relations), transcribe the decompiled pseudocode into z3 almost verbatim:
from z3 import *
flag = [BitVec(f'c{i}', 8) for i in range(32)]
s = Solver()
# constraints copied from the decompilation, e.g.:
s.add((flag[i] ^ 0x37) + i & 0xff == target[i])
s.add(And(*[And(0x20 <= c, c < 0x7f) for c in flag])) # printable
s.check(); m = s.model(); print(bytes(m[c].as_long() for c in flag))
Use BitVec (not Int) so overflow wraps like machine arithmetic. If the decompile is messy,
feed the whole function to angr (angr with find=<success addr>, avoid=<fail addr>) and let
it symbolically execute to the answer.
- Know the recurring shapes. Flag = input satisfying equations (z3); keygen (invert the
serial transform, then generate any valid one); VM challenges (small custom bytecode — write a
30-line disassembler for the handlers, the flag falls out); wasm (wasm2c/wabt, then same game);
mobile crackmes (
mobile-apk-reverse).
Gotchas
- The first "Correct" string can be a decoy — verify your answer actually prints success on
the real binary; some challenges have fake success paths guarding the real check.
- Anti-debug that crashes you is the solve path — if debugging changes behavior, run under
the debugger anyway and note where; the flag transformation sometimes only happens under ptrace.
- BitVec vs Int — z3
Int doesn't wrap; forgetting & 0xff semantics makes solvers return
unsat on trivially solvable checks.
- Multiple valid flags — constraint checks can accept many inputs; the scoreboard wants the
one in flag format, so add
flag{-prefix constraints when known.
- Don't statically invert what you can dynamically read — a
strcmp in ltrace beats an hour
of decompilation; try dynamic first, always.
- Stripped/optimized binaries mislead decompilers — check the disassembly when pseudocode
looks impossible (
reverse-eng-binary-triage for triage discipline).
Verify success
The recovered input, entered into the original unmodified binary, produces the success path —
and matches the event flag format. For keygens, two independently generated keys both validate.
References
Triage: reverse-eng-binary-triage; deep analysis: reverse-eng-ghidra, reverse-eng-ida,
reverse-eng-radare2; unpacking/anti-analysis: reverse-eng-deobfuscation; z3 docs; angr.
Triage via ctf-methodology.
1---2name: ctf-rev3description: >4 CTF reversing speed-run — crack flag-check binaries fast: locate the check via strings/xrefs,5 bypass anti-debug and obfuscation, and solve constraint-checkers with z3 instead of manual6 algebra. Load when the handout is an ELF/PE/exe that asks for input or a key. Signals: "rev"7 category, "Enter the flag/key/password", correct/wrong messages, crackme, packed binary,8 anti-debug (ptrace, IsDebuggerPresent), keygen challenge.9domain: ctf10type: technique11stability: learning12modes: [pentest, bugbounty]13severity: medium14cwe: [CWE-798]15tools: [radare2, ghidra, ida-pro, gdb, ltrace, strace, z3, angr, upx, strings]16schema_version: 117metadata:18 source: https://skillmd.com/skills/noorqureshi/ctf-rev19---2021# CTF reverse engineering speed-run2223## When it applies24The handout is a binary that checks something — a flag, key, serial, or password — and you need25the accepted input. This skill is the CTF *workflow*: find the check fast, defang whatever stops26you from seeing it, and solve the constraints mechanically. For deep tool use defer to27`reverse-eng-binary-triage` (first-pass recon), `reverse-eng-radare2` / `reverse-eng-ghidra` /28`reverse-eng-ida` (analysis), `reverse-eng-deobfuscation` (unpacking/devirtualization).2930## Why it works31A flag-checker is a pure function from input to accept/reject, and CTF ones are small. You never32need to understand the whole binary — only the check — so the game is *locating* it (strings and33xrefs beat linear reading), *exposing* it (anti-debug and packing are thin in CTF), and *solving*34it (the constraints are almost always z3-shaped; manual algebra is the slow path).3536## Method371. **Two-minute triage.** `file`, `strings -n 6` (+ `strings -el` for UTF-16), `checksec`, run it38 with junk input. The strings that matter: "Correct!"/"Wrong", the flag format itself39 (sometimes just… present), odd alphabets, and packer markers (`UPX!`). Packed → `upx -d` first;40 anything fancier → `reverse-eng-deobfuscation`. .NET/Java/Go/Rust/Python-exe binaries route to41 their dedicated tooling (`reverse-eng-dotnet`, `reverse-eng-go-rust`, pyinstxtractor+uncompyle).422. **Locate the check, don't read the binary.** In r2/Ghidra/IDA: find the "Wrong" string, follow43 the xref to its function, and read *backwards* from the branch that prints it. Everything44 between the input read and that branch is the check. For C/C++ this is usually one function.453. **Dynamic shortcuts before static grinding.**46 - `ltrace ./chall` — library calls like `strcmp(input, "...")`, `strlen`, per-char compares47 leak the answer or its length with zero analysis.48 - Compare-per-character binaries → count crashes/coverage: flip one input byte at a time and49 watch which comparison index changes, or use a debugger on the compare loop to read each50 expected byte as it's checked.51 - `strace` for flag files, env vars, network the binary secretly consults.524. **Defang anti-debug (it's shallow in CTF).** Common tricks and one-line counters:53 - `ptrace(PTRACE_TRACEME)` → patch the call, or `catch syscall ptrace` in gdb and force54 return 0.55 - `IsDebuggerPresent` / `PEB.BeingDebugged` → zero the flag in the debugger, or patch the56 conditional jump that follows.57 - Timing checks (`rdtsc`) → patch or ignore — they only guard a branch.58 - Self-checksum / anti-tamper → make *all* your patches before the checksum computes, or patch59 the comparison result, not the data.60 The universal move: find the check that *branches on* the anti-debug result and flip that one61 branch — don't fight each mechanism.625. **Solve the check with z3 — the CTF superpower.** When the check transforms input bytes and63 compares against constants (xor/add/rotate/substitute chains, matrix math, per-position64 relations), transcribe the decompiled pseudocode into z3 almost verbatim:65 ```python66 from z3 import *67 flag = [BitVec(f'c{i}', 8) for i in range(32)]68 s = Solver()69 # constraints copied from the decompilation, e.g.:70 s.add((flag[i] ^ 0x37) + i & 0xff == target[i])71 s.add(And(*[And(0x20 <= c, c < 0x7f) for c in flag])) # printable72 s.check(); m = s.model(); print(bytes(m[c].as_long() for c in flag))73 ```74 Use `BitVec` (not `Int`) so overflow wraps like machine arithmetic. If the decompile is messy,75 feed the whole function to angr (`angr` with `find=<success addr>, avoid=<fail addr>`) and let76 it symbolically execute to the answer.776. **Know the recurring shapes.** Flag = input satisfying equations (z3); keygen (invert the78 serial transform, then generate any valid one); VM challenges (small custom bytecode — write a79 30-line disassembler for the handlers, the flag falls out); wasm (wasm2c/wabt, then same game);80 mobile crackmes (`mobile-apk-reverse`).8182## Gotchas83- **The first "Correct" string can be a decoy** — verify your answer actually prints success on84 the real binary; some challenges have fake success paths guarding the real check.85- **Anti-debug that crashes you *is* the solve path** — if debugging changes behavior, run under86 the debugger anyway and note where; the flag transformation sometimes only happens under ptrace.87- **BitVec vs Int** — z3 `Int` doesn't wrap; forgetting `& 0xff` semantics makes solvers return88 `unsat` on trivially solvable checks.89- **Multiple valid flags** — constraint checks can accept many inputs; the scoreboard wants the90 one in flag format, so add `flag{`-prefix constraints when known.91- **Don't statically invert what you can dynamically read** — a `strcmp` in ltrace beats an hour92 of decompilation; try dynamic first, always.93- **Stripped/optimized binaries mislead decompilers** — check the disassembly when pseudocode94 looks impossible (`reverse-eng-binary-triage` for triage discipline).9596## Verify success97The recovered input, entered into the *original unmodified* binary, produces the success path —98and matches the event flag format. For keygens, two independently generated keys both validate.99100## References101Triage: `reverse-eng-binary-triage`; deep analysis: `reverse-eng-ghidra`, `reverse-eng-ida`,102`reverse-eng-radare2`; unpacking/anti-analysis: `reverse-eng-deobfuscation`; z3 docs; angr.103Triage via `ctf-methodology`.