Reversing Network Protocols

Reversing Network Protocols

An undocumented binary protocol is reverse-engineered from two sides at once: the wire, which shows you the bytes that actually flow, and the client, which shows you the code that produced them. Neither alone is enough — the capture tells you what varies, the client tells you why — and the deliverable is a parser precise enough to decode, replay, and eventually fuzz the protocol.

When to Use

• A proprietary or custom TCP/UDP protocol with no public specification
• Traffic Wireshark displays as raw Data bytes because no dissector matches
• Game, IoT, industrial, or C2 protocols you must decode from captures plus the client binary
• Building a Wireshark dissector, a Kaitai Struct spec, or a scapy layer to parse and replay a protocol

When NOT to Use

• A known/structured protocol — Protobuf or gRPC specifically is attacking-grpc-protobuf; standard protocols have dissectors already.
• Defensive PCAP investigation — hunting, IOC extraction, incident triage — is analyzing-network-traffic.
• Reversing the client binary itself (disassembly, decompilation) is analyzing-binaries; do that in service of the protocol, then structure it here.
• Identifying or breaking the crypto once you find the protocol is encrypted — reviewing-cryptography.

Work Both Sides

From the capture, establish structure by comparing many messages:

• Framing — how a message knows where it ends: a length prefix (a field that tracks payload size across messages), a delimiter, or fixed-size records. Finding the length field is usually the first breakthrough.
• Constants and magic — bytes identical across every message mark headers, version fields, or type tags.
• Opcode / message type — a small field that correlates with different message shapes; group captures by it.
• Counters and sequence numbers — fields that increment monotonically.
• Checksums — a trailing field that changes unpredictably with the payload; test CRC variants against the message body.
• TLV — many custom protocols are type-length-value triplets once you see the pattern.
• Endianness — confirm from a known length: does a 260-byte message carry 04 01 or 01 04?

From the client, resolve what the capture cannot:

• Find the serialization/parsing code by reversing the binary (analyzing- binaries), and read how it builds and consumes a message.
• Hook send/recv (or the app's socket wrapper) with Frida to capture the buffer before encryption and after decryption — this is how you read an encrypted protocol without breaking the crypto.
• If the protocol is encrypted, hook the plaintext side; identify the cipher and key handling with reviewing-cryptography only if you must operate off-client.

Build a Parser

Turn the recovered structure into something executable, because a parser is both the proof you understood the protocol and the tool for everything after:

• Wireshark dissector (Lua) to decode live captures field by field — the fastest way to validate a hypothesis against more traffic.
• Kaitai Struct to describe the binary format declaratively and generate parsers in several languages.
• scapy custom layers to both parse and craft messages for replay and fuzzing.

Iterate: decode a batch, find the message that does not parse, refine the spec. The malformed message is where your model is wrong, not noise.

Then What

A working parser enables the security work: replay to test whether the server validates sequence, session, and authentication; fuzz individual fields (lengths, type tags, counts) to find parser bugs; and reason about protocol-level auth — whether nonces, session tokens, or MACs actually prevent replay and tampering, or are decorative.

Rationalizations to Reject

• "Wireshark shows it as raw data, so there's nothing to see." No dissector matched — that is the starting point, not a dead end. Diff messages to find framing and fields, or write a dissector.
• "It's encrypted, so I can't reverse it." Hook the client's send/recv to read plaintext before encryption. You rarely need to break the cipher to understand the protocol.
• "I'll just eyeball the hex." Structure emerges from comparison across many messages — the field that tracks length, the byte that selects type. One message in isolation hides all of it.
• "The parser mostly works, one message fails — close enough." The failing message is the counterexample that corrects your model. Chase it; it is where the real structure is.
• "Replaying it is harmless." Replay against a live service acts on that service. Treat it as testing that needs authorization, and reason about what a replayed message does before sending it.

References

• analyzing-binaries — reversing the client to read its serialization code
• attacking-grpc-protobuf — when the protocol is Protobuf/gRPC, not custom
• analyzing-network-traffic — defensive PCAP investigation, not protocol RE
• reviewing-cryptography — identifying and assessing the protocol's crypto