Scapy packet, field, and layer patterns

Scapy packet, field, and layer patterns

Use this skill when working on protocol implementation in Scapy core/layers, especially around Packet, Field, fields_desc, layer binding, and regression tests via UTScapy.

Core model

• A layer is a Packet subclass with a fields_desc list.
• Field values flow through human/internal/machine conversions (h2i, i2h, i2m, m2i, any2i).
• Building and dissecting are centralized in Packet (do_build, post_build, do_dissect, guess_payload_class, extract_padding).

Packet and field usage patterns

1) Define layers as Packet subclasses with fields_desc

• Prefer explicit defaults in fields_desc.
• Use field types that encode semantics (e.g. EnumField, FlagsField, PacketListField) instead of generic numeric/string fields when possible.
• For optional/variant fields, use ConditionalField and MultipleTypeField.

Examples:

• scapy/layers/inet.py (IP, TCP, UDP, ICMP)
• scapy/layers/l2.py (Ether, ARP, GRE, Dot1Q)

2) Compute deferred values in post_build

• Keep None defaults for values that must be computed from final bytes (checksums, lengths, header offsets).
• Implement updates in post_build(self, p, pay) after payload is available.

Common examples:

• IP.post_build: IHL/len/checksum
• TCP.post_build: data offset + checksum
• UDP.post_build: len + checksum
• GRE.post_build: conditional checksum

3) Split payload/padding with extract_padding when length is explicit

• If a layer encodes payload length, override extract_padding.
• Return (payload, padding) correctly to keep dissection aligned.

Examples:

• IP.extract_padding
• UDP.extract_padding
• Dot3.extract_padding

4) Payload dispatch and layer binding (guess_payload_class, bind_*)

• Packet.guess_payload_class(payload) selects the next layer during dissection.
  • Default behavior iterates payload_guess entries and matches field constraints.
  • payload_guess is populated by bind_bottom_up() (and therefore by bind_layers()).
  • If no match applies, fallback is Packet.default_payload_class() (conf.raw_layer).
• Override guess_payload_class only when dispatch depends on dynamic logic that cannot be expressed as simple field equality.
• Base-class dispatch pattern: a base layer class can decode into a concrete sibling/subclass before normal dissection.
  • Commonly implemented with dispatch_hook(cls, _pkt, ...).
  • Example: Ether.dispatch_hook can return Dot3, and Dot3.dispatch_hook can return Ether.
• Binding APIs:
  • bind_bottom_up(lower, upper, ...): dissection-time binding only.
  • bind_top_down(lower, upper, ...): build-time default field propagation only.
  • bind_layers(lower, upper, ...): convenience helper that applies both directions.

Examples:

• scapy/packet.py: guess_payload_class, bind_bottom_up, bind_top_down, bind_layers
• scapy/layers/l2.py: Ether.dispatch_hook / Dot3.dispatch_hook
• scapy/layers/inet.py: protocol-specific guess_payload_class overrides

5) Request/response matching and stream reassembly hooks

• Implement hashret() when a protocol needs stable request/response correlation keys.
  • Base Packet.hashret() delegates to payload; many layers override it (e.g. IP/TCP/UDP) to include flow context.
• Implement answers(other) to express protocol-level "is response to" logic.
  • Base behavior matches same class then delegates to payload.
  • Typical checks include type/code pairs, id/seq tuples, and src/dst or sport/dport relationships.
• For stream-aware sniffing:
  • IPSession performs on-the-fly IPv4 defragmentation.
  • TCPSession reconstructs TCP byte streams and can call layer tcp_reassemble(data, metadata, session) when implemented.
  • tcp_reassemble should return a packet when enough bytes are present, else None and keep state in metadata / session.

References:

• scapy/packet.py (hashret, answers)
• scapy/layers/inet.py (IP/TCP/UDP/ICMP overrides)
• scapy/sessions.py (IPSession, TCPSession, streamcls)

6) Complete scapy.fields field-type catalog (and intended use)

This list covers field types defined in scapy/fields.py that are used when authoring packet layouts.

• Base and wrappers (core composition/control)
  • Field: base class for conversion and build/dissect behavior.
  • Emph: display emphasis wrapper.
  • MayEnd: allows legal early stop while dissecting optional tail fields.
  • ActionField: triggers side-effect callbacks during value assignment/build.
  • ConditionalField: include/exclude field based on runtime predicate.
  • MultipleTypeField: pick one of several field definitions from packet context.
  • PadField, ReversePadField: alignment/padding wrappers.
  • TrailerField, FCSField: model trailer fields appended after main payload.
• Address/network identity fields
  • DestField, MACField, LEMACField: MAC/address-like values.
  • IPField, SourceIPField: IPv4 values (including context-derived source behavior).
  • IP6Field, SourceIP6Field, DestIP6Field: IPv6 values.
  • IPPrefixField, IP6PrefixField: address + prefix-length encoding.
  • OUIField: 24-bit organization identifiers.
  • UUIDField, UUIDEnumField: UUID values (optionally enumerated).
• Integer scalar fields
  • Byte-sized: ByteField, XByteField, OByteField, SignedByteField, YesNoByteField
  • 3-byte and variable-width: ThreeBytesField, X3BytesField, LEThreeBytesField, XLE3BytesField, NBytesField, XNBytesField
  • 16-bit: ShortField, SignedShortField, LEShortField, LESignedShortField, XShortField, XLEShortField
  • 32-bit: IntField, SignedIntField, LEIntField, LESignedIntField, XIntField, XLEIntField
  • 64-bit: LongField, SignedLongField, LELongField, LESignedLongField, XLongField, XLELongField
• Floating/scaled numeric fields
  • IEEEFloatField, IEEEDoubleField: IEEE-754 float/double.
  • BCDFloatField: BCD-encoded decimal values.
  • FixedPointField: fixed-point values stored in bitfields.
  • ScalingField, BitScalingField: physical-unit scaling (offset/resolution) wrappers.
• String/bytes fields
  • Generic strings: StrField, StrFieldUtf16
  • Enumerated strings: StrEnumField
  • Fixed-length strings: StrFixedLenField, StrFixedLenFieldUtf16, StrFixedLenEnumField, NetBIOSNameField
  • Length-coupled strings: StrLenField, StrLenFieldUtf16, StrLenEnumField, BoundStrLenField
  • Hex-rendered byte strings: XStrField, XStrLenField, XStrFixedLenField, XLEStrLenField
  • Terminated strings: StrNullField, StrNullFieldUtf16, StrStopField
• Packet/container and length-coupling fields
  • Nested packets: PacketField, PacketLenField, PacketListField
  • Generic repeated values: FieldListField
  • Length/count references: FieldLenField, LEFieldLenField, LenField
• Bit and varint-like fields
  • BitField, BitLenField, BitFieldLenField, XBitField
  • BitExtendedField, LSBExtendedField, MSBExtendedField
• Enum-capable fields
  • Generic and char: EnumField, CharEnumField
  • Bit enums: BitEnumField, BitLenEnumField
  • Width/endianness-specific enums: ShortEnumField, LEShortEnumField, LongEnumField, LELongEnumField, ByteEnumField, XByteEnumField, IntEnumField, SignedIntEnumField, LEIntEnumField, XLEIntEnumField, XShortEnumField, LE3BytesEnumField, XLE3BytesEnumField
  • Contextual enums: MultiEnumField, BitMultiEnumField
  • Enum-key variants: ByteEnumKeysField, ShortEnumKeysField, IntEnumKeysField
• Flags fields
  • FlagsField: bitmask rendered as named flags.
  • MultiFlagsField: multi-set flag representation across grouped masks.
• Time fields
  • UTCTimeField: timestamps with human-readable UTC conversion support.
  • SecondsIntField: integer-seconds timestamp fields.
• Raw override
  • RawVal (from scapy.packet) is used with fields when you intentionally bypass normal conversion.

Examples:

• scapy/layers/inet.py: IP.options (PacketListField), ICMP conditional/enum usage
• scapy/layers/l2.py: ARP MultipleTypeField usage, GRE conditional/checksum fields

7) Additional protocol implementation patterns (from Scapy implementation slides)

• Layer navigation while debugging/building packets
  • pkt[Layer] selects a specific layer instance.
  • .underlayer accesses the previous (encapsulating) layer.
  • .payload accesses the next layer.
  • Useful to validate binding/dispatch behavior in quick REPL checks.
• Field-family naming shortcuts
  • X* variants: hex-oriented display.
  • LE* variants: little-endian integer encoding.
  • Signed* variants: signed integer representation.
  • *Enum* variants: symbolic names via mapping tables.
• PacketField / PacketListField dynamic decoding
  • PacketField decodes one embedded packet object.
  • PacketListField decodes repeated embedded packets; dispatcher may be a class or a function returning a class.
  • Common with TLV/IE lists where element type depends on runtime bytes.
• Post-dissection mutation hooks
  • post_dissect(self, s) can adjust decoded state after full layer parsing (e.g., decrypting payload-dependent content).
  • Use only when mutation is required after field extraction.
• Explicit payload vs. extra-bytes control
  • extract_padding can intentionally discard/redirect trailing bytes when they are not protocol padding. This is important when parsing variable-length lists of embedded packets where trailing bytes should not be interpreted as padding.
• Custom field implementation workflow
  • Model field states explicitly:
    • internal (i): Scapy runtime value
    • machine (m): serialized bytes representation
    • human (h): display representation
  • Override conversion/encoding points as needed:
    • i2h, i2m, m2i for state conversion
    • addfield / getfield for custom serialization and parsing behavior

References:

• scapy/contrib/mpls.py (guess_payload_class dispatch pattern)
• scapy/layers/inet6.py (IPv46.dispatch_hook)
• scapy/contrib/automotive/doip.py (tcp_reassemble)
• scapy/layers/dot11.py (Dot11WEP.post_dissect)
• scapy/layers/tftp.py (TFTP_Option.extract_padding)

UTScapy integration patterns

Use UTScapy for regression coverage of layer behavior.

Campaign structure

• Campaign syntax in test files:
  • % campaign
  • + test set
  • = unit test
  • ~ keywords
  • * comments
• The last Python expression in a unit test determines pass/fail truthiness.

References:

• doc/scapy/development.rst (Testing with UTScapy)
• scapy/tools/UTscapy.py (parse_campaign_file, campaign execution/filtering)

Useful UTScapy CLI patterns

• Run one or more campaigns: -t
• Include/exclude keyword groups: -k / -K
• Select tests by number: -n
• Load .utsc JSON config: -c
• Output formats: -f text|ansi|HTML|LaTeX|xUnit|live
• Generate docs from campaign comments/tests: -R
• Non-root mode keyword filtering: -N

Reference:

• scapy/tools/UTscapy.py (usage(), main())

Typical test workflow

1. Add/modify protocol layer fields and binding logic.
2. Add/adjust UTScapy tests in test/ configs/campaign files with meaningful keywords.
3. Run through existing wrappers (./test/run_tests or tox environments using scapy.tools.UTscapy).
4. Keep tests focused on:
   • build/dissect roundtrips
   • computed fields (len/checksum/options)
   • payload dispatch and edge-case fallback

Quick references

• Core packet behavior: scapy/packet.py
• Field internals: scapy/fields.py
• Common layer patterns: scapy/layers/inet.py, scapy/layers/l2.py
• Test runner internals: scapy/tools/UTscapy.py
• Design guidance: doc/scapy/build_dissect.rst