Custom protocol reverse engineering
When it applies
The target speaks something that isn't plain HTTP/JSON: a custom binary TCP/UDP protocol,
Protobuf/gRPC with no reflection, FlatBuffers/MessagePack, WebSocket or MQTT frames, or a
private RPC format — and you have a PCAP, a proxy export, client logs, or the client binary.
(For pure HTTP parameter signing in JS, use reverse-eng-js; for gRPC's HTTP/2 attack surface
itself, api-grpc.) Target and any replay testing must be in scope.txt
(tradecraft-scope-roe) — replay only against systems you're authorized to touch.
Why it works
Protocols are machines, and machines are regular: fixed headers, magic bytes, length fields,
monotonic sequence numbers, and repeated type-length-value structures. Align many samples of
the same message type and the invariant bytes identify themselves; the variable ones are your
fields. Serialization formats like Protobuf are self-describing enough to decode blind — field
numbers and wire types survive even without the .proto.
Method
- Capture and triage. Pull raw payloads and label each sample by direction (C→S / S→C);
note handshake, heartbeat, and reconnect patterns:
tshark -r cap.pcap -Y "tcp.port==4433" -T fields -e frame.number -e ip.src -e tcp.payload | head
First questions: fixed header? magic bytes? a length field? TLV or fixed-size records? Any
compression (zlib/gzip/lz4) or per-frame encryption (AES/ChaCha)?
- Recover the frame layout. Align several messages of the same kind and diff the bytes:
invariant regions = header/constants; counters = sequence numbers; a field whose value
matches the remaining byte count = length (check endianness, and whether it includes the
header). Locate integrity fields last — CRC16/32, checksums, HMAC slots. Sketch the state
machine (
Connect → Auth → Ready → Request/Response → Close). Encode the layout as a
Wireshark Lua dissector, an ImHex/010 Editor template, or a Kaitai Struct .ksy.
- Tackle serialization and crypto. Protobuf:
protoc --decode_raw < msg.bin for a quick
read, blackboxprotobuf or pbtk to rebuild a usable .proto. gRPC is HTTP/2 headers plus
a protobuf body. Encrypted frames: the key derivation lives in the client — pull it with
reverse-eng-binary-triage (native), reverse-eng-js (web), or mobile tooling; look for the
nonce/IV adjacent to the ciphertext.
- Produce the artifacts. A message-type table (name / opcode / fields), at least one
reproducible decode command or script, and evidence excerpts (raw hex + decoded result,
redacted of secrets and third-party data).
See cheatsheet.md for the layout-pattern table, parser skeleton, and tshark one-liners.
Gotchas
- Length-field ambiguity — big- vs little-endian, and header-inclusive vs body-only, are
the two classic off-by-entire-parse errors; validate against a known sample's byte count.
- A crash on replay is not proof of a bug — it may be a checksum/sequence check rejecting
you; fix your framing first, then judge the response.
- Compressed-before-encrypted vs encrypted-before-compressed — high entropy over the whole
body means encryption; compression headers (
\x78\x9c, \x1f\x8b) mean you can decompress
directly.
- Don't fuzz blind against the live service — replay a captured, harmless message first;
mutate one field at a time, and only within scope.
Verify success
Your dissector or script decodes a fresh capture into the correct fields with no manual
fix-ups, and you can state the message-type dictionary and state machine. Bonus proof: a
replayed (in-scope) message with one mutated field is accepted by the server — the layout is
right, and the protocol is now fuzzable.
References
Wireshark/tshark docs; Kaitai Struct gallery; blackboxprotobuf; api-grpc, web-websocket,
reverse-eng-binary-triage in this library.
Portions adapted from reverse-skill by zhaoxuya520, MIT License.
1---2name: reverse-eng-protocol3description: >4 Reverse a custom network protocol from captured traffic or the client binary: recover the5 frame layout, message-type dictionary, field meanings, and state machine. Load for custom6 TCP/UDP binary protocols, Protobuf/gRPC without reflection, FlatBuffers/MessagePack,7 WebSocket/MQTT/private RPC framing, PCAP-driven format recovery, length-prefixed or TLV8 frames, magic bytes, CRC/checksum fields, or encrypted frame headers.9domain: reverse-engineering10type: technique11stability: learning12modes: [pentest, bugbounty]13severity: medium14cwe: [CWE-319]15tools: [tshark, wireshark, python3, blackboxprotobuf, kaitai-struct, imhex]16schema_version: 117metadata:18 source: https://skillmd.com/skills/noorqureshi/reverse-eng-protocol19---2021# Custom protocol reverse engineering2223## When it applies24The target speaks something that isn't plain HTTP/JSON: a custom binary TCP/UDP protocol,25Protobuf/gRPC with no reflection, FlatBuffers/MessagePack, WebSocket or MQTT frames, or a26private RPC format — and you have a PCAP, a proxy export, client logs, or the client binary.27(For pure HTTP parameter signing in JS, use `reverse-eng-js`; for gRPC's HTTP/2 attack surface28itself, `api-grpc`.) Target and any replay testing must be in `scope.txt`29(`tradecraft-scope-roe`) — replay only against systems you're authorized to touch.3031## Why it works32Protocols are machines, and machines are regular: fixed headers, magic bytes, length fields,33monotonic sequence numbers, and repeated type-length-value structures. Align many samples of34the same message type and the invariant bytes identify themselves; the variable ones are your35fields. Serialization formats like Protobuf are self-describing enough to decode blind — field36numbers and wire types survive even without the `.proto`.3738## Method391. **Capture and triage.** Pull raw payloads and label each sample by direction (C→S / S→C);40 note handshake, heartbeat, and reconnect patterns:41 ```bash42 tshark -r cap.pcap -Y "tcp.port==4433" -T fields -e frame.number -e ip.src -e tcp.payload | head43 ```44 First questions: fixed header? magic bytes? a length field? TLV or fixed-size records? Any45 compression (zlib/gzip/lz4) or per-frame encryption (AES/ChaCha)?462. **Recover the frame layout.** Align several messages of the same kind and diff the bytes:47 invariant regions = header/constants; counters = sequence numbers; a field whose value48 matches the remaining byte count = length (check endianness, and whether it includes the49 header). Locate integrity fields last — CRC16/32, checksums, HMAC slots. Sketch the state50 machine (`Connect → Auth → Ready → Request/Response → Close`). Encode the layout as a51 Wireshark Lua dissector, an ImHex/010 Editor template, or a Kaitai Struct `.ksy`.523. **Tackle serialization and crypto.** Protobuf: `protoc --decode_raw < msg.bin` for a quick53 read, `blackboxprotobuf` or `pbtk` to rebuild a usable `.proto`. gRPC is HTTP/2 headers plus54 a protobuf body. Encrypted frames: the key derivation lives in the client — pull it with55 `reverse-eng-binary-triage` (native), `reverse-eng-js` (web), or mobile tooling; look for the56 nonce/IV adjacent to the ciphertext.574. **Produce the artifacts.** A message-type table (name / opcode / fields), at least one58 reproducible decode command or script, and evidence excerpts (raw hex + decoded result,59 redacted of secrets and third-party data).6061See [cheatsheet.md](cheatsheet.md) for the layout-pattern table, parser skeleton, and tshark one-liners.6263## Gotchas64- **Length-field ambiguity** — big- vs little-endian, and header-inclusive vs body-only, are65 the two classic off-by-entire-parse errors; validate against a known sample's byte count.66- **A crash on replay is not proof of a bug** — it may be a checksum/sequence check rejecting67 you; fix your framing first, then judge the response.68- **Compressed-before-encrypted vs encrypted-before-compressed** — high entropy over the whole69 body means encryption; compression headers (`\x78\x9c`, `\x1f\x8b`) mean you can decompress70 directly.71- **Don't fuzz blind against the live service** — replay a captured, harmless message first;72 mutate one field at a time, and only within scope.7374## Verify success75Your dissector or script decodes a fresh capture into the correct fields with no manual76fix-ups, and you can state the message-type dictionary and state machine. Bonus proof: a77replayed (in-scope) message with one mutated field is accepted by the server — the layout is78right, and the protocol is now fuzzable.7980## References81Wireshark/tshark docs; Kaitai Struct gallery; blackboxprotobuf; `api-grpc`, `web-websocket`,82`reverse-eng-binary-triage` in this library.8384---85_Portions adapted from [reverse-skill](https://github.com/zhaoxuya520/reverse-skill) by zhaoxuya520, MIT License._