VXLAN Packet Format

Chapter Overview

Understanding the detailed VXLAN packet format is essential for network analysis, troubleshooting, and optimization. This section provides an in-depth look at every field in a VXLAN packet.

Complete Packet Structure

A VXLAN packet consists of multiple nested headers that enable Layer 2 transport over Layer 3 networks:

Outer Ethernet Header (14 bytes)
  • Destination MAC (6 bytes): Next-hop MAC address
  • Source MAC (6 bytes): Local VTEP MAC address
  • EtherType (2 bytes): 0x0800 (IPv4) or 0x86DD (IPv6)
0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Destination MAC Address                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Destination MAC Address    |     Source MAC Address       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Source MAC Address                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        EtherType              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Outer IP Header (20 bytes IPv4 / 40 bytes IPv6)
  • Version (4 bits): IP version (4 or 6)
  • IHL (4 bits): Header length (IPv4 only)
  • ToS/DSCP (8 bits): Type of Service
  • Total Length (16 bits): Total packet length
  • Source IP: Local VTEP IP address
  • Destination IP: Remote VTEP IP address
IPv4 Header:
0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version|  IHL  |Type of Service|          Total Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Identification        |Flags|      Fragment Offset    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Time to Live |    Protocol   |         Header Checksum       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Source Address                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Destination Address                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
UDP Header (8 bytes)
  • Source Port (16 bits): Ephemeral port for ECMP
  • Destination Port (16 bits): 4789 (VXLAN)
  • Length (16 bits): UDP header + payload length
  • Checksum (16 bits): Optional error detection
0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Length             |           Checksum            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
VXLAN Header (8 bytes)
  • Flags (8 bits): Control flags
  • Reserved (24 bits): Must be zero
  • VNI (24 bits): VXLAN Network Identifier
  • Reserved (8 bits): Must be zero
0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|R|R|R|I|R|R|R|            Reserved                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                VXLAN Network Identifier (VNI) |   Reserved    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Inner Ethernet Header (14 bytes)
  • Destination MAC (6 bytes): Original destination
  • Source MAC (6 bytes): Original source
  • EtherType (2 bytes): Inner frame type
0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Destination MAC Address                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Destination MAC Address    |     Source MAC Address       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Source MAC Address                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        EtherType              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

VXLAN Flags Field

The VXLAN flags field contains control information:

Bit Name Description Value
0 R Reserved 0
1 R Reserved 0
2 R Reserved 0
3 R Reserved 0
4 I VNI Present 1
5-7 R Reserved 0

VXLAN Flags Example

# Standard VXLAN Header
Flags: 0x08 (00001000 binary)
- Bit 4 (I flag) = 1 (VNI Present)
- All other bits = 0 (Reserved)

# Invalid VXLAN Header
Flags: 0x10 (00010000 binary)
- Bit 3 = 1 (Should be 0)
- Bit 4 (I flag) = 0 (Should be 1)
- Invalid packet, should be dropped

UDP Source Port Calculation

The UDP source port is crucial for ECMP load balancing:

Hash Calculation

Source port calculated from inner frame fields:

  • Inner source MAC address
  • Inner destination MAC address
  • Inner IP addresses (if present)
  • Inner TCP/UDP ports (if present)
ECMP Benefits

Proper source port selection enables:

  • Equal utilization of parallel paths
  • Consistent flow hashing
  • Optimal bandwidth utilization
  • Reduced network congestion

Packet Size Considerations

VXLAN encapsulation significantly increases packet size:

Component IPv4 Size IPv6 Size Notes
Outer Ethernet 14 bytes 14 bytes Same for both versions
Outer IP 20 bytes 40 bytes IPv6 header larger
UDP 8 bytes 8 bytes Same for both versions
VXLAN 8 bytes 8 bytes Same for both versions
Total Overhead 50 bytes 70 bytes Significant overhead

MTU Impact Example

# Standard Ethernet Frame
Original MTU: 1500 bytes
VXLAN Overhead: 50 bytes (IPv4)
Effective MTU: 1450 bytes

# Jumbo Frame
Original MTU: 9000 bytes  
VXLAN Overhead: 50 bytes (IPv4)
Effective MTU: 8950 bytes

# Recommendation: Enable jumbo frames on underlay

Packet Example

Complete VXLAN packet with sample values:

VXLAN Packet Analysis

# Outer Ethernet Header
Dst MAC: 00:1A:2B:3C:4D:5E (Next hop router)
Src MAC: 00:AA:BB:CC:DD:EE (VTEP1 interface)
Type:    0x0800 (IPv4)

# Outer IP Header  
Version: 4
IHL:     5 (20 bytes)
ToS:     0x00
Length:  124 bytes (total packet)
ID:      0x1234
Flags:   0x4000 (Don't Fragment)
TTL:     64
Protocol: 17 (UDP)
Checksum: 0x5678
Src IP:  192.168.1.1 (VTEP1)
Dst IP:  192.168.1.2 (VTEP2)

# UDP Header
Src Port: 49152 (ephemeral)
Dst Port: 4789 (VXLAN)
Length:   96 bytes
Checksum: 0x9ABC

# VXLAN Header
Flags:    0x08 (VNI Present)
Reserved: 0x000000
VNI:      0x002774 (10100)
Reserved: 0x00

# Inner Ethernet Header
Dst MAC: 00:11:22:33:44:55 (VM1)
Src MAC: 00:AA:BB:CC:DD:FF (VM2)
Type:    0x0800 (IPv4)

# Inner IP Payload
IPv4 packet from VM2 to VM1

Wireshark Analysis

Analyzing VXLAN packets with Wireshark:

Wireshark Filters

# Filter VXLAN traffic
vxlan

# Filter specific VNI
vxlan.vni == 10100

# Filter VXLAN with inner IP
vxlan and ip.addr == 192.168.100.1

# Filter VXLAN encapsulation issues
vxlan.flags != 0x08

# Filter VXLAN by outer VTEP
ip.src == 192.168.1.1 and udp.port == 4789

Common Issues

  • Invalid Flags: VNI flag not set (0x08)
  • Wrong UDP Port: Not using port 4789
  • MTU Problems: Fragmentation due to overhead
  • Checksum Errors: Hardware offload issues
Previous: Scalability Benefits Next: Encapsulation Process

Encapsulation Process

Chapter Overview

The VXLAN encapsulation process transforms original Ethernet frames into VXLAN packets. Understanding this process is crucial for troubleshooting and optimization.

Encapsulation Overview

VXLAN encapsulation occurs at the ingress VTEP when frames need to be transported over the IP network:

Frame Ingress
→
VLAN to VNI Mapping
→
Destination Lookup
→
Header Addition
→
Transmission

Step-by-Step Process

The encapsulation process involves several steps:

Step 1: Frame Reception

VTEP receives Ethernet frame from local segment

  • Frame arrives on access port
  • VLAN tag identified
  • Source MAC learning
  • Broadcast domain determination
Step 2: VLAN to VNI Mapping

Determine VNI based on ingress VLAN

  • Lookup VLAN to VNI mapping
  • Validate VNI membership
  • Apply network policies
  • Set QoS markings
Step 3: Destination Lookup

Find destination VTEP for target MAC

  • MAC table lookup
  • Local vs remote determination
  • VTEP IP address resolution
  • Flooding decision if unknown
Step 4: Header Addition

Add VXLAN encapsulation headers

  • VXLAN header construction
  • UDP header with port 4789
  • IP header with VTEP addresses
  • Outer Ethernet header

Encapsulation Example

# Step 1: Frame Reception
Interface: Ethernet1/1
VLAN: 100
Frame: Host A → Host B
Original Size: 1500 bytes

# Step 2: VLAN to VNI Mapping
VLAN 100 → VNI 10100
Policy: Allow
QoS: DSCP 0

# Step 3: Destination Lookup
Destination MAC: 00:11:22:33:44:55
Lookup Result: Remote VTEP 192.168.1.2
Tunnel: VTEP1 → VTEP2

# Step 4: Header Addition
VXLAN Header: VNI 10100, Flags 0x08
UDP Header: Src 49152, Dst 4789
IP Header: 192.168.1.1 → 192.168.1.2
Ethernet Header: Next-hop MAC addresses
Final Size: 1550 bytes

VNI Determination

VNI assignment can be based on different criteria:

Method Description Configuration Use Case
VLAN-based Map VLAN ID to VNI vlan X vn-segment Y Traditional migration
Port-based Map access port to VNI interface X vni Y Simplified access
Policy-based Dynamic VNI assignment Policy rules Automated provisioning
MAC-based Map MAC address to VNI mac-vni table Fine-grained control

Destination Resolution

The VTEP must determine where to send the encapsulated frame:

Known Destination

MAC address found in forwarding table

  1. Lookup destination MAC in VNI table
  2. Retrieve remote VTEP IP address
  3. Encapsulate with specific VTEP IP
  4. Send unicast to remote VTEP
Unknown Destination

MAC address not in forwarding table

  1. Identify as BUM traffic
  2. Determine VNI flooding scope
  3. Use multicast or replication
  4. Send to all VTEPs in VNI

UDP Source Port Generation

The UDP source port is critical for ECMP load balancing:

Source Port Calculation Methods

# Method 1: Inner Frame Hash
hash_input = inner_src_mac + inner_dst_mac + inner_vlan
source_port = hash(hash_input) % 16384 + 49152

# Method 2: Inner IP Hash  
hash_input = inner_src_ip + inner_dst_ip + inner_protocol
source_port = hash(hash_input) % 16384 + 49152

# Method 3: 5-tuple Hash
hash_input = inner_src_ip + inner_dst_ip + inner_protocol + 
             inner_src_port + inner_dst_port
source_port = hash(hash_input) % 16384 + 49152

# Port Range: 49152-65535 (RFC 6335)

ECMP Considerations

Proper source port selection ensures:

  • Consistent flow hashing across network devices
  • Equal distribution of traffic across ECMP paths
  • Optimal bandwidth utilization
  • Reduced risk of link congestion

QoS Preservation

VXLAN encapsulation can preserve QoS markings:

QoS Field Inner Location Outer Location Preservation Method
DSCP Inner IP ToS Outer IP ToS Copy or map
CoS Inner VLAN tag Outer VLAN tag Copy or map
Traffic Class Inner IPv6 Outer IPv6 Copy or map

Hardware Acceleration

Modern hardware provides VXLAN encapsulation acceleration:

ASIC Acceleration
  • Hardware-based encapsulation
  • Line-rate performance
  • Parallel processing
  • Dedicated lookup engines
NIC Offload
  • SmartNIC processing
  • CPU offload
  • Reduced system load
  • Higher throughput
Software Optimization
  • Vectorized processing
  • Batch operations
  • Memory optimization
  • Cache efficiency

Troubleshooting Encapsulation

Common encapsulation issues and troubleshooting approaches:

Debug Commands

# Enable encapsulation debugging
debug nve packet encap

# Show encapsulation statistics
show nve interface nve1 detail

# Show VLAN to VNI mapping
show vlan vn-segment

# Show MAC learning
show l2route evpn mac all

# Monitor encapsulation counters
show interface nve1 counters

Common Issues

  • Missing VNI Mapping: VLAN not mapped to VNI
  • MAC Learning Failure: Unable to resolve destination
  • MTU Problems: Encapsulation exceeds MTU
  • QoS Mapping: Incorrect DSCP/CoS handling
  • Hardware Limits: ASIC table exhaustion

Best Practices

  • Use consistent VNI allocation schemes
  • Implement proper QoS mapping
  • Monitor encapsulation performance
  • Enable hardware acceleration when available
  • Configure appropriate MTU sizes
Previous: Packet Format Next: Decapsulation Process