VXLAN DATA PLANE

This section describes the traffic forwarding process on the VXLAN forwarding plane.

1. Traffic model

Depending on the traffic flow direction and scope, DCN traffic can be classified into east-west traffic (transmitted within a DC) and north-south traffic (sent across the DC). The DCN traffic additionally falls into four types, with Figure 6.12 showing the distributed network overlay traffic model.

• • Traffic transmitted within the same subnet of a VPC is forwarded by a TOR switch after Layer 2 VXLAN encapsulation.
• • Traffic transmitted between subnets of the same VPC is forwarded by a TOR switch based on Layer 3 routes. This is done after Layer 3 VXLAN encapsulation. ^[1]

FIGURE 6.12 Four traffic models on the distributed network overlay.

In Figure 6.13, VM_A, VM_B, and VM_C are all on the network segment 10.1.1.0/24 and belong to VNI 5000. At this point, VM_A needs to communicate with VM_C.

Due to first time communication, VM_A does not have VM_C’s MAC address. Therefore, VM_A broadcasts an ARP Request packet, requesting VM_C’s MAC address.

1. ARP Request packet forwarding process

FIGURE 6.13 Communication on the same subnet between VMs.

Figure 6.14 shows the ARP Request packet forwarding process.

a. VM_A broadcasts an ARP Request packet, requesting VM_C’s MAC address. The source MAC address of the packet is MAC_A, the destination MAC address is all Fs, the source IP address is IP_A, and the destination IP address is IP_C.

b. After VTEP_1 receives the ARP Request packet, it determines that the packet is required to go through the VXLAN tunnel based on the device configuration. After identifying the BD to which the packet belongs, VTEP_1 further identifies the VNI to which the packet belongs. VTEP_1 learns

FIGURE 6.14 ARP Request packet forwarding process

the entry of MAC_A, VNI, and inbound interface (Port_l), saving the entry to the local MAC address table. Afterward, VTEP_1 replicates the packet based on the ingress replication list, carrying out packet encapsulation.

In the encapsulated packets, the following occurs: the outer source IP address is the local VTEP’s (VTEP_1) IP address, the outer destination IP address is the remote VTEPs’ (VTEP_2 and VTEP_3) IP addresses, and the outer source MAC address is the local VTEP’s MAC address. Lastly, the outer destination MAC address is the next-hop device’s MAC address, which heads toward the destination IP network.

Following the completion of encapsulation, the packet is transmitted on the IP network according to the outer MAC and IP addresses, until it arrives at the remote VTEP.

c. After the packets arrive at VTEP_2 and VTEP_3, the VTEPs decapsulate the packets to obtain the original packets sent by VM_A. VTEP_2 and VTEP_3 each learn the entry of VM_A’s MAC address, VNI, and remote VTEP’s IP address (IP_1), saving the entry to the local MAC address table. Next, VTEP_2 and VTEP_3 process the packets according to the device configuration and broadcast them in the corresponding Layer 2 domain.

After VM_B and VM_C receive the ARP Request packet, they check if the destination IP address matches the local host IP address. VM_B then discovers that the destination IP address is not the local IP address, discarding the packet. VM_C finds that the destination IP address is the local IP address and responds to the ARP Request packet.

2. ARP Reply packet forwarding process

Figure 6.15 shows the ARP Reply packet forwarding process.

a. Because VM_C has already learned VM_A’s MAC address at this point, the ARP Reply packet is a unicast packet. The source MAC address of the packet is MAC_C, the destination MAC address is MAC_A, the source IP address is IP_C, and the destination IP address is IP_A.

b. After VTEP_3 receives the ARP Reply packet from VM_C, it identifies the VN to which the packet belongs. (The identification process is similar to step 2 of the ARP Request packet.) VTEP_3 learns the entry of MAC_C, VNI, and inbound interface (Port_3), and saves the entry to the local MAC address table. Then, VTEP_3 encapsulates the packet.

The outer source IP address is the IP address of the local VTEP (VTEP_3), and the outer destination IP address is the IP address of the remote VTEP (VTEP_1). The outer source MAC address is the MAC address of the local VTEP, and the outer destination MAC address is the MAC address of the next-hop device on the destination IP network.

FIGURE 6.15 ARP Reply packet forwarding process.

After encapsulation, the packet is transmitted on the IP network according to the outer MAC and IP addresses, until it arrives at the remote VTEP.

c. After the packet arrives at VTEP_1, VTEP_1 decapsulates the packet to obtain the original packet sent by VM_C. VTEP_1 learns the entry of VM_C’s MAC address, VNI, and remote VTEP’s IP address (IP_3), and saves the entry to the local MAC address table. VTEP_1 then decapsulates the packet and sends it to VM_A.

Up until this point, VM_A and VM_C have already learned each other’s MAC address. After that point, VM_A and VM_C will communicate instead in unicast mode. The encapsulation and decapsulation processes of unicast packets are similar to those shown in Figure 6.15.

3. Intra-subnet forwarding of BUM packets

Intra-subnet BUM packets are forwarded only between Layer 2 VXLAN gateways and are unknown to Layer 3 VXLAN gateways. Intra-subnet BUM packets can be forwarded in ingress replication mode.

In ingress replication mode, after a BUM packet enters a VXLAN tunnel, the ingress VTEP performs VXLAN encapsulation based on the ingress replication list and sends the packet to all the egress VTEPs in the list. When the BUM packet leaves the VXLAN tunnel, the egress VTEPs decapsulate it.

Figure 6.16 shows the forwarding process of a BUM packet in ingress replication mode. Terminal A is connected to the distributed gateway Leafl and is required to send BUM traffic to the VXLAN network.

• • After Leafl receives a packet from Terminal A, it determines the Layer 2 BD of the packet, based on the access interface and VLAN ID in the packet.
• • The VTEP on Leafl obtains the tunnel list for the VNI based on the BD and replicates the packet based on the ingress replication list. It then performs VXLAN tunnel encapsulation, before forwarding it to the outbound interface.
• • After the VTEP on Leaf 2 or Leaf 3 receives the VXLAN packet, it checks the UDP destination port number, source and destination IP addresses, and VNI of the packet to determine the packet validity. Leaf2 or Leaf3 obtains the Layer 2 BD based on the VNI and performs VXLAN decapsulation to obtain the inner Layer 2 packet. ^[2]

FIGURE 6.16 Forwarding process of an intra-subnet BUM packet in ingress replication mode.

4. Inter-subnet packet forwarding

Inter-subnet packets must be forwarded through a Layer 3 gateway. Figure 6.17 shows the inter-subnet packet forwarding process in distributed VXLAN gateway scenarios. Hostl and Host2 belong to different subnets and are required to communicate with each other.

FIGURE 6.17 Inter-subnet packet forwarding.

The following describes the forwarding process. For details about the ARP forwarding process, refer to the process of forwarding known unicast packets (including the ARP Request/Reply packet processing) on the same subnet. ¹

1. After Leafl receives a packet from Hostl, it finds that the destination MAC address of the packet is a gateway MAC address. Therefore, this packet must be forwarded at Layer 3.

• 2. Leafl determines the Layer 2 BD of the packet based on the inbound interface and accordingly locates the L3VPN instance bound to the VBDIF interface of the Layer 2 BD. Leafl then searches the L3VPN routing table and locates the destination address of packet. Figure 6.18 shows the host route in the L3VPN routing table. Leafl obtains the Layer 3 VNI and next-hop address of the host route, and finds that the recursive outbound interface is a VXLAN tunnel interface. Therefore, Leafl determines that the packet must be transmitted through a VXLAN tunnel.
• - Leafl obtains MAC addresses based on the VXLAN tunnel’s source and destination IP addresses, replacing the source and destination MAC addresses in the inner Ethernet header.
• - Leafl encapsulates the packet with the Layer 3 VNI.
• - Leafl encapsulates the VXLAN tunnel’s source and destination IP addresses in the outer IP header. The MAC address of the NVE1 interface is the source MAC address, and the next- hop MAC address is the destination MAC address used in the outer Ethernet header.
• 1. The VXLAN packet is then transmitted over the IP network based on the IP and MAC addresses in the outer headers, and finally reaches Leaf2.
• 2. After Leaf2 receives the VXLAN packet, it decapsulates the packet and finds that the destination MAC address is its own MAC address. Hence, the packet must be forwarded at Layer 3.

FIGURE 6.18 Host route 1 in the L3VPN routing table.

FIGURE 6.19 Host route 2 in the L3VPN routing table.

3. Leaf2 finds the L3VPN instance based on the Layer 3 VNI carried in the packet, searches the routing table of the L3VPN instance (as shown in Figure 6.19), and obtains the gateway interface address as the next hop of the packet. It then replaces the destination MAC address with the MAC address of Host2, replaces the source MAC address with Leaf2’s MAC address, and forwards the packet to Host2.

Host2 sends packets to Hostl in the same process.

5. Inter-VPC packet forwarding

Figure 6.20 shows the access process. VM1 and VM3 are deployed on the same or different physical servers (identical process) and are connected to the same or different vSwitches. VM1 and VM3 belong to compute nodes in different VPCs. VM1 initiates access to VM3, and traffic needs to be filtered by the firewalls of the two VPCs because the traffic is transmitted across VPCs.

The following describes the forwarding process. For details related to the ARP process, refer to the process of forwarding known unicast packets on the same subnet (including the ARP Request/Reply packet processing).

• VM1 sends an ARP Request packet to request for the MAC address of the local network segment’s gateway. ^•

After NVE1 receives the ARP Request packet, it sends an ARP Reply packet to VM1 in place of the gateway.

• • VM1 sends the first data packet to VM3.
• • NVE1 receives the first data packet and finds that the destination address does not belong to the network segment of VRF- A. The packet matches the default route in VRF-A and is then sent to VRF-A on the service leaf node. The two distributed

FIGURE 6.20 Cross-VPC packet forwarding on the distributed overlay.

VRFs (VRF-A) on NVE1 and the service leaf node exchange information through a VXLAN overlay tunnel (Layer 3 VNI interconnection).

• The data packet of VM1 matches the default route in VRF-A on the service leaf node and is forwarded to vSys-A of the firewall. The firewall is connected to the service leaf node through a VLAN, and hence, the packet is forwarded as a common Ethernet packet. The firewall searches for the route for VPC communication in vSys-A and then forwards the packet to vSys-B. The firewall searches for the route in vSys-B and finally forwards the packet to VRF-B of the service leaf node.

• • The service leaf node searches for the host route of VM3 in VRF-B and forwards the packet to VRF-B of NVE2 through the VXLAN tunnel (Layer 3 VNI interconnection). NVE2 searches for the host route, removes the VXLAN tag from the packet, and forwards the packet to VM3.
• • The forwarding process for packets sent from VM3 to VM1 is identical to above.
• 6. Forwarding of internal and external traffic in a DC

Figure 6.21 depicts the forwarding and access process of internal and external traffic in a DC. VM1 is a compute node in a VPC of the

FIGURE 6.21 Distributed VXLAN traffic leaving a DC.

DCN and needs to access an IP address of a network, such as the Internet, outside the DC.

The following describes the forwarding process. For details related to the ARP process, refer to the process of forwarding known unicast packets on the same subnet (including the ARP Request/Reply packet processing).

• 1. VM1 sends an ARP Request packet to request the MAC address of the gateway on the local subnet.
• 2. After NVE1 receives the ARP Request packet, it sends an ARP Reply packet to VM1 in place of the gateway.
• 3. VM1 sends the first data packet to the public IP address on the Internet.
• 4. After receiving the first data packet, N VE1 finds that the destination address is not in the network segment of VRF-A and then sends the data packet to VRF-A on the service leaf node through the default route in VRF-A. The two distributed VRFs (VRF-A) interwork through a VXLAN overlay tunnel (Layer 3 VNI interconnection).
• 5. The data packet of VM1 matches the default route in VRF-A on the service leaf node and is forwarded to vSys-A of the firewall. The firewall is connected to the service leaf node through a VLAN, and hence, the packet is forwarded as a common Ethernet packet.
• 6. The firewall searches for the route in vSys-A and forwards the data packet to vSys-root. In addition, the firewall translates the source address of the data packet. vSys-root of the firewall forwards the data packet to the public vRouter of the service leaf node through the default route.
• 7. The public vRouter of the service leaf node forwards the data packet to the public vRouter of the border leaf node through the default route. The public vRouters on the service leaf node and border leaf node are connected through Layer 3 VNIs.
• 8. The border leaf node forwards packets through the underlay network to the VRF on the PE connecting to Internet links.
• 9. The PE forwards the packet from the DC.

• [1] Traffic transmitted between VPCs is forwarded across subnets,and isolation for security purposes is required. Therefore, tomeet this, the traffic needs to pass through a firewall and reachthe Layer 3 VXLAN gateway. • Traffic sent from a user outside the DC to a server in a VPCpasses through the Intrusion Prevention System (IPS) or firewall,LB, VXLAN gateway, and TOR switch before reaching the server. The forwarding plane forwards known unicast packets and BUMpackets. This occurs on intra-subnet packets, inter-subnet packets,cross-VPC packets, and packets inside and outside a DC. The following describes each scenario. 2. Intra-subnet known unicast packet forwarding (including ARPRequest/Reply packet processing)
• [2] Leaf2 or Leaf3 checks the destination MAC address of the innerLayer 2 packet and finds it is a BUM MAC address. Therefore,Leaf2 or Leaf3 broadcasts the packet onto the network connected to terminals (not the VXLAN tunnel side) in the Layer2 BD. Specifically, Leaf2 or Leaf3 finds the outbound interfacesand encapsulation information unrelated to the VXLAN tunnel,adds VLAN tags to the packet, and finally forwards the packet toTerminal В or Terminal C.