CTF Challenge Triage and Solve Methodology
Authorized-use caveat: This skill is for legal CTF competitions, security education, and authorized research environments only. Never apply these techniques to systems without explicit written authorization.
Phase 0 — Initial Triage (always first)
Identify challenge category before touching any tool.
# Universal file fingerprint
file challenge_file
xxd challenge_file | head -4 # magic bytes
strings -n 8 challenge_file | head -30
binwalk challenge_file # embedded files / compression layers
exiftool challenge_file # metadata (images, PDFs)
| Magic bytes / extension | Category | Next phase |
|---|---|---|
| ELF / PE / Mach-O | Pwn / Rev | Phase 1a / 1b |
| PNG / JPG / BMP / WAV | Steganography / Forensics | Phase 2 |
| PCAP / PCAPNG | Network forensics | Phase 3 |
Memory dump (.raw, .vmem, .dmp) |
Memory forensics | Phase 4 |
| Ciphertext / hex blob / base64 | Crypto | Phase 5 |
| Web URL / source | Web | Use exploit/web/ skills |
| Archive / unknown binary | Misc / forensics | Phase 6 |
Phase 1a — Pwn (Binary Exploitation)
Step 1 — Security property enumeration
checksec --file=./binary
# Key outputs: RELRO, Stack Canary, NX, PIE, RUNPATH
| Property | Absent = exploitable via... |
|---|---|
| Stack canary | Stack overflow → ret2libc / ROP |
| NX (No-Execute) | Shellcode injection to stack/heap |
| PIE | Fixed address assumptions for ROP gadgets |
| Full RELRO | GOT overwrite |
Step 2 — Static analysis
objdump -d ./binary | grep -A 20 "<main>"
strings ./binary | grep -E "(flag|CTF|pass|secret|/bin/sh)"
readelf -s ./binary | grep -E "(sym|FUNC)"
# Heavy analysis
ghidra ./binary # or: r2 -A ./binary
Step 3 — Dynamic analysis + offset finding
# GDB with GEF/PEDA
gdb -q ./binary
# Inside GDB:
# pattern create 200
# run < <(python3 -c "print('A'*200)")
# pattern offset $rsp
# Pwntools skeleton
python3 - <<'EOF'
from pwn import *
context.binary = elf = ELF('./binary')
p = process('./binary')
# or: p = remote('challenge.host', 1337)
offset = cyclic_find(0x61616161) # replace with crashing value
payload = b'A' * offset + p64(elf.sym['win'])
p.sendline(payload)
p.interactive()
EOF
Step 4 — ROP chain (when NX set)
ROPgadget --binary ./binary --rop | grep "pop rdi"
# or: ropper -f ./binary --search "pop rdi"
python3 - <<'EOF'
from pwn import *
elf = ELF('./binary')
libc = ELF('./libc.so.6')
rop = ROP(elf)
rop.call('puts', [elf.got['puts']])
rop.call(elf.sym['main'])
# Leak libc base, then ret2libc
EOF
Step 5 — Libc identification (remote exploits)
# Identify libc from leaked addresses
python3 -m one_gadget libc.so.6
# or: libc-database lookup
Phase 1b — Reverse Engineering
Step 1 — Static disassembly
# Strings for quick wins (flag format, hardcoded keys)
strings ./binary | grep -iE "(flag\{|ctf\{|[A-Z0-9_]{10,}\})"
# Ghidra headless (no GUI)
analyzeHeadless /tmp/ghidra_proj ChalProj \
-import ./binary \
-postScript PrintASM.java \
-scriptPath /opt/ghidra/Ghidra/Features/Decompiler/ghidra_scripts \
2>/dev/null
# radare2 (faster for known binary formats)
r2 -A ./binary
# Inside r2: afl (functions), pdf @ sym.main, VV (visual)
Step 2 — Anti-debug / packer detection
# Detect packers
upx -t ./binary # UPX packed?
die ./binary # Detect-It-Easy
# If packed: upx -d ./binary
# Entropy analysis (high entropy sections = packed/encrypted)
binwalk -E ./binary
If packed, see reverser/packer-unpacking/SKILL.md.
Step 3 — Symbolic execution for constraint solving
# Angr — solve unknown input to reach target state
python3 - <<'EOF'
import angr, claripy
proj = angr.Project('./binary', auto_load_libs=False)
flag_chars = [claripy.BVS(f'flag_{i}', 8) for i in range(32)]
flag = claripy.Concat(*flag_chars)
state = proj.factory.full_init_state(stdin=flag)
for c in flag_chars:
state.add_constraints(c >= 0x20, c <= 0x7e)
sm = proj.factory.simulation_manager(state)
sm.explore(find=lambda s: b'Correct' in s.posix.dumps(1),
avoid=lambda s: b'Wrong' in s.posix.dumps(1))
if sm.found:
print(sm.found[0].solver.eval(flag, cast_to=bytes))
EOF
Phase 2 — Steganography / Image Forensics
Triage order (fast to slow)
# 1. Metadata
exiftool challenge.png
# Look for: Comment field, GPS coords, Software, hidden IPTC/XMP data
# 2. Appended data after EOF marker
xxd challenge.jpg | tail -20
# JPEG ends at FF D9; anything after = appended content
# 3. LSB steganography (most common CTF technique)
zsteg challenge.png # PNG/BMP LSB
stegsolve challenge.png # GUI — bit plane analysis (run with: java -jar stegsolve.jar)
steghide extract -sf challenge.jpg -p "" # try empty passphrase first
steghide extract -sf challenge.jpg -p "password"
# 4. Outguess (JPEG)
outguess -r challenge.jpg output.txt
# 5. Audio steganography
sox challenge.wav -n stat # audio properties
spectral view in Audacity or Sox spectrogram
Password brute-force for steghide
stegseek challenge.jpg /usr/share/wordlists/rockyou.txt
Phase 3 — Network Forensics (PCAP)
# Quick summary
capinfos challenge.pcap
tshark -r challenge.pcap -q -z io,phs # protocol hierarchy
tshark -r challenge.pcap -q -z conv,tcp # TCP conversations
# Extract HTTP objects (images, files)
tshark -r challenge.pcap --export-objects http,/tmp/pcap_http/
# Extract credentials
tshark -r challenge.pcap -Y "ftp || http.request.method==POST" -T fields \
-e frame.number -e ip.src -e tcp.payload
# Follow TCP stream (stream index from Wireshark or tshark)
tshark -r challenge.pcap -q -z follow,tcp,ascii,0
# DNS exfiltration
tshark -r challenge.pcap -Y dns -T fields -e dns.qry.name | sort -u | grep -v "\.arpa"
Phase 4 — Memory Forensics
# Identify OS profile
vol3 -f memory.raw windows.info 2>/dev/null || vol3 -f memory.raw linux.info
# Windows
vol3 -f memory.raw windows.pslist
vol3 -f memory.raw windows.cmdline
vol3 -f memory.raw windows.netscan
vol3 -f memory.raw windows.malfind # injected code
vol3 -f memory.raw windows.dumpfiles --pid <PID> --output-dir /tmp/
# Linux
vol3 -f memory.raw linux.pslist
vol3 -f memory.raw linux.bash
# Carve files
foremost -i memory.raw -o /tmp/foremost_out/
Phase 5 — Cryptography Challenges
Step 1 — Identify cipher / encoding
# Encoding layers (base64, hex, rot13, etc.)
echo "encoded_string" | base64 -d
echo "encoded_string" | xxd -r -p # hex to binary
echo "encoded_string" | tr 'A-Za-z' 'N-ZA-Mn-za-m' # ROT13
# CyberChef magic function: https://gchq.github.io/CyberChef/#recipe=Magic(3,false,false,'')
# Hash identification
hash-identifier <hash>
hashid <hash>
Step 2 — Classic cipher analysis
# Frequency analysis (substitution ciphers)
python3 - <<'EOF'
from collections import Counter
ct = "YOUR_CIPHERTEXT_HERE"
freq = Counter(c for c in ct.upper() if c.isalpha())
for char, count in freq.most_common(10):
print(f"{char}: {count} ({count/len([c for c in ct if c.isalpha()])*100:.1f}%)")
# English: E=12.7%, T=9%, A=8.2%, O=7.5%
EOF
Step 3 — RSA attacks
# Factor small/weak modulus
python3 - <<'EOF'
from sympy import factorint
n = <modulus>
factors = factorint(n)
print(factors) # p, q
# If factored: phi=(p-1)*(q-1), d=pow(e,-1,phi), m=pow(c,d,n)
EOF
# Wiener's attack (small private exponent)
# RsaCtfTool covers common attacks:
python3 RsaCtfTool.py --publickey pub.pem --attack all --uncipherfile cipher.bin
# Common primes / known factors
# Check factordb.com for n
Step 4 — Hash cracking
hashcat -m <mode> hash.txt /usr/share/wordlists/rockyou.txt
# -m 0=MD5, 1000=NTLM, 1800=sha512crypt, 13000=RAR5
john --wordlist=/usr/share/wordlists/rockyou.txt --format=<format> hash.txt
Phase 6 — Misc / Archive Analysis
# Multi-layer extraction
binwalk -eM challenge_file # recursive extraction
7z l archive.7z # list contents without extracting
zip2john archive.zip > zip.hash && john zip.hash
# QR codes / barcodes
zbarimg image.png
# or: zxing online decoder
# PDF analysis
pdfinfo document.pdf
pdf-parser.py -o 1 document.pdf # extract object
peepdf document.pdf -i # interactive analysis
CTF Solve Workflow Summary
1. file + strings + xxd (30 seconds) → category decision
2. Category-specific triage (Phases 1-6 above)
3. Quick wins first: hardcoded strings, empty passphrase, default creds
4. If stuck > 15 min: try adjacent technique (encoding layer, nested file)
5. Note flag format from challenge description (e.g., FLAG{...}, ctf{...})
6. Validate flag matches expected format before submitting
ATT&CK Mapping
| Phase | Technique |
|---|---|
| Binary exploitation | T1059.004 (Unix Shell), T1203 (Exploit for Client Exec) |
| Packer analysis | T1027.002 (Obfuscated Files - Software Packing) |
| Crypto decoding | T1140 (Deobfuscate/Decode Files), T1027 (Obfuscated Files) |
| Stego extraction | T1564.003 (Hidden in Files/Images) |
| Memory forensics | T1055 (Process Injection detection), T1070 (Indicator Removal) |
Common CTF Pitfalls
- Flag encoding: The flag may be base64/hex-encoded inside the file — always run
stringsfirst - Wrong endianness: x86 is little-endian; addresses printed by pwntools are auto-handled, but manual math is not
- ASLR vs PIE: ASLR randomizes the stack/heap; PIE randomizes the binary base. Leak a pointer before building ROP chains
- Steghide vs zsteg: steghide works on JPEG/BMP with a passphrase; zsteg targets PNG/BMP with LSB patterns
- Nested archives: binwalk
-eM(recursive) is essential — CTF files routinely nest 3-4 compression layers - Unicode / non-ASCII in crypto: Check for zero-width characters, homoglyphs, whitespace encoding