Cutting-Edge Technologies

This chapter gives a glimpse into several new technologies in the DC field, including the container, hybrid cloud, and AI Fabric technologies. Upon finishing this chapter, you will have an understanding of these technologies’ basic concepts, existing technical solutions in the industry, application scenarios, and development trends.

CONTAINER

Overview

Container technology is an OS virtualization technology that provides an isolated OS environment based on the shared OS kernel of a host. Refer to Figure 11.1 for an overview of the container technology architecture. Flere, you can see that each container contains the following components: independent file system namespace (MNT NS), host and domain namespace (UTS NS), inter-process communication namespace (IPC NS), process ID namespace (PID NS), network protocol stack namespace (NET NS), and user namespace (user NS). In addition, a container is mounted to a vSwitch through a vNIC, and the vSwitch accesses a network through a pNIC.

Container technology is developed in two phases: standalone container and container cluster. In the former, a variety of OS isolation technologies have been developed based on the UNIX or Linux OS, thereby laying the foundation for the virtualization technology of a single container.

FIGURE 11.1 Container technology architecture.

Leveraging these isolation technologies, the UNIX or Linux OS is able to provide an isolated OS environment. The following describes major OS isolation technologies:

• • chroot: also called chroot jail, was first developed on UNIX version 7 in 1979. chroot provides the independent file system space for each process by changing the root directory of a process and its child processes to a specific file system directory, as well as restricting the access of the process and its child processes to only the specific directory.
• • FreeBSD jail: was developed by R&D Associates, a hosting provider, in 2000 to provide an independent and clean service running environment for customers on the shared FreeBSD system. FreeBSD jail can partition a FreeBSD-derived compute system into several independent mini-systems, which are isolated from each other, as well as having an independent file system and independent common users, administrators, and network space. ^[1]

• • Open Virtuozzo (OpenVZ): an open-source container platform released by Virtuozzo in 2005. OpenVZ is also called VPS or virtual environment. Each OpenVZ contains an independent file system, users and user groups, process trees, network devices, and IPC objects.
• • Linux containers (LXC): an open-source container engine based on CGroup and Linux NameSpace on an open-source Linux system. In contrast to FreeBSD jail, Solaris Containers, and OpenVZ, which are based on UNIX, LXC is developed based on the Linux kernel and can be easily promoted in the industry.
• • Docker: was originally an internal project initiated by Solomon Hykes, dotCloud’s founder, and was released as open source in 2013. In Docker 0.9 and versions earlier than Docker 0.9, LXC is used as the container execution engine. In versions later than Docker 0.9, Libcontainer is used as the execution engine. Based on Linux CGroup and namespace technologies, Docker provides container image packaging and management, as well as container lifecycle management.
• • Kubernetes: an open-source container orchestration system developed by Google. It provides container cluster management, container cluster scaling, and inter-host communication for containers through CNIs. Currently, Kubernetes is managed by the Linux Foundation.
• • Docker Swarm: a container orchestration system released by Docker in 2014. It provides the following functions: Docker cluster management, container cluster scaling, service discovery, load balancing, and inter-host container network communication. In contrast to Kubernetes, Docker Swarm cannot flexibly support user-defined interfaces and has limited openness.
• • Mesos: was first released in 2009 as a distributed cluster management platform. Then, with the development of container technologies, Mesos started to support container clusters in 2014. Mesos is a larger platform than Kubernetes and Docker Swarm, making it suitable for a super-large cluster system (with more than 100,000 containers).

Due to that fact that containers are more flexible and efficient than VMs, software applications are gradually being migrated from VMs to containers. Container cluster technologies provide numerous advantages, such as solving the single point of failures of standalone containers, providing unified orchestration capabilities of compute, storage, and network resources for large-scale container applications, and accelerating the development of container technologies. Major container platforms include Google’s Kubernetes, Docker’s Docker Swarm, and Mesos at the University of California, Berkeley, CA, USA. Of these, Kubernetes has, over years of development, become the de facto standard for container platforms.

11.1.2 Industry's Mainstream Container Network Solutions Figure 11.2 shows the typical three-layer architecture of container-based web, apps, and DBs. A client accesses the frontend LB (ingress) cluster through a URL. Then, an LB Pod cluster performs Layer 7 load balancing based on service types and forwards requests to the service cluster of the web frontend. The Pod of the web cluster then accesses the Kubernetes service of the backend app cluster based on the service logic. After NAT and load balancing have been performed on the Kubernetes services, the actual service traffic is converted into the east-west traffic to be directly exchanged between Pods.

Similarly, when the Pod of the app cluster accesses the DB cluster service, the apps access the DB service. Then, using the Kubernetes service, NAT

FIGURE 11.2 Typical three-layer architecture of web, apps, and DBs.

and LB service traffic is converted into east-west traffic to be exchanged between Pods.

According to the preceding service scenarios, the Kubernetes network has the following typical requirements:

• • The network that carries east-west traffic between Pods must be reachable, and east-west load balancing or NAT must be supported.
• • Mutual access between services and Pods, north-south load balancing, and clients’ access to container clusters must be supported.

To meet these requirements, the Kubernetes community defines three principles for the Kubernetes network:

• • All containers can communicate with all other containers without NAT.
• • All nodes can communicate with all containers (and vice versa) without NAT.
• • The IP that a container sees itself as is the same IP that others see it as. The following example refers to Figure 11.3 to describe the three principles:
• • Podl can directly access Pod2, Pod3, and Pod4 without NAT.
• • Podl can directly access nodel and node2 without NAT.
• • nodel can directly access Podl, Pod2, Pod3, and Pod4 without NAT.

The Kubernetes community defines basic principles for direct IP access between Pods in a cluster and between Pods and hosts. It clearly defines east-west network requirements, without defining specific network solutions. Currently, common open-source container network solutions include Contiv, Calico, Flannel, and Pluawei CNI-Genie, which can be classified into three types: host overlay solution, network overlay Layer 2 bridging solution, and network overlay Layer 3 routing solution. ^[2]

FIGURE 11.3 Three principles of the Kubernetes network.

this architecture, a vSwitch on a physical server functions as a Pod gateway and VTEP to connect to the overlay or underlay network. Inter-Pod traffic on the same physical server is directly forwarded by a vSwitch, whereas Inter-Pod traffic on different physical servers is forwarded through the VXLAN tunnel between vSwitches. Open- source or container platform vendors automate container overlay network provisioning by providing Kubernetes API server watcher and Kubernetes CNI plugins.

Flannel, a typical host overlay solution in the industry, does not provide IP address allocation, which instead needs to be implemented by a third-party component (ETCD), as shown in Figure 11.5. On the control plane, Flannel uses ETCD to configure a routing table on each node. On the data plane, it uses the tunnel mode (such as VXLAN and GRE) for traffic forwarding. Flannel runs a daemon process called Flanneld on each host. When a host is added to a

FIGURE 11.5 Flannel host overlay solution.

cluster, Flanneld applies for an IP subnet from ETCD and obtains IP subnet information of other hosts.

Because each host uses a different IP subnet, the container’s IP address will be changed if a container is migrated to another host. Flannel overcomes this by using the Pod as the basic unit for network management. That is, IP addresses are allocated per Pod, while subnets are allocated per node (host or VM). Specifically, all Pods on a node must use the same subnet, and a Pod cannot be deployed across nodes.

The following describes the implementation of the Flannel host overlay solution:

• • Flannel of a Kubernetes node registers its information (subnets
• 10.0. 9.0/24 and 10.0.19.0/24) with ETCD.
• • The Kubernetes node periodically queries network information from ETCD. If a subnet is added, the node updates the subnet and IP address information in the local routing table.
• • When a Kubernetes container needs to access the subnet
• 10.0. 19.0/24, it sends a packet destined for 10.0.19.2 and routes the packet to the local VXLAN tunnel for encapsulation and forwarding based on its local routing table. After the destination Kubernetes node receives the VXLAN-encapsulated packet, it decapsulates the packet and forwards it to the destination Pod based on its local routing table.

The Flannel host overlay solution implements the Kubernetes networking model and allows Pods to communicate across Kubernetes nodes. However, when a large amount of data is transmitted in highly concurrent cluster scenarios, vSwitch forwarding capabilities can hold back the overall performance. Such a bottleneck can be eliminated by deploying a network overlay solution based on the Kubernetes open-source architecture, as it uses hardware switches instead of vSwitches to encapsulate and decapsulate VXLAN packets.

2. Network overlay Layer 2 bridging solution

Figure 11.6 shows the typical architecture of the network overlay Layer 2 bridging solution. In this architecture, hardware switches on a fabric network function as Pod gateways and VTEPs, while Pods

FIGURE 11.6 Typical architecture of the network overlay Layer 2 bridging solution.

connect to the overlay network through VLANs. Traffic between Pods on the same physical server is exchanged through a vSwitch, whereas that between Pods on different physical servers is forwarded through VXLAN at Layer 3. To complete server-side network provisioning of containers, open-source or container platform vendors provide Kubernetes API server watcher and Kubernetes CNI plugins.

When it comes to Layer 2 bridging, Contiv is a typical solution in the industry. It includes the Netmaster (Kubernetes API server watcher in Figure 11.6) deployed on the Kubernetes master node, as well as Netplugins (Kubernetes CNI plugins) deployed on Kubernetes nodes, as shown in Figure 11.7. Of these, Netmaster monitors the Kubernetes API server, allocates VLANs and IP addresses to Pods, and stores the information in ETCD. Each Netplugin implements

FIGURE 11.7 Contiv Layer 2 bridging solution.

the CNI networking model used by Kubernetes. Contiv uses JSON- RPC to distribute endpoints from Netplugin to Netmaster, while Netplugin handles Up/Down events from Pods and configures vSwitches.

3. Network overlay Layer 3 routing solution

Figure 11.8 shows the typical architecture of the network overlay Layer 3 routing solution. In this architecture, a Pod gateway is deployed on a vRouter of a physical server, and a hardware switch functions as a VTEP to connect the physical server to the overlay network. EBGP is used by the physical server to advertise Pod routes to the overlay VRF. Traffic between Pods on the same physical server is exchanged through vRouters, whereas that between Pods on different physical servers is forwarded through VXLAN at Layer 3. In this process, a hardware switch provides a large number of VXLAN integrated routing and bridging (IRB) routes, whereas open-source or PaaS vendors provide Kubernetes API server watcher and Kubernetes CNI plugins to implement Pod network provisioning and routing configuration on container servers.

FIGURE 11.8 Typical architecture of the network overlay Layer 3 routing solution.

Calico is a typical solution in the industry for implementing open-source Layer 3 routing. It includes the Calico controller, Felix, and BGP client, which can be seen in Figure 11.9. Calico controller (Kubernetes API server watcher) monitors the API server; detects the events of services, Pods, and ingresses; allocates IP addresses; and stores the IP addresses in ETCD. A BGP client, implemented using the bird Internet routing daemon (BIRD) in an open-source BGP project, implements BGP, advertises routes of the local Pod through an RR, and learns Pod routes on other Kubernetes nodes. A vRouter is implemented based on the routing function of the Linux OS. Felix, a Kubernetes CNI plugin, is deployed on each Kubernetes node to listen to ETCD, detect the events of Pods and services, create Pod vNICs, configure IP addresses for Pods, and update the Linux kernel routing table and security policies.

FIGURE 11.9 Calico open-source Layer 3 routing solution.

Table 11.1 provides a comparison between the three container network solutions: host overlay solution, network overlay Layer 2 bridging solution, and network overlay Layer 3 routing solution. In the host overlay solution, the container network is completely decoupled from the infrastructure layer, and the forwarding performance is poor. Such characteristics make this solution suitable for scenarios where the PaaS platform is independently deployed. The network overlay Layer 2 bridging solution is easy to deploy and provides high forwarding performance but supports only limited deployment scale, making it applicable to scenarios with a low container virtualization ratio (for example, database scenarios). The network overlay Layer 3 routing solution provides high forwarding performance and

• [1] Solaris Containers: also called Solaris Zones, is an independent virtual server on the Solaris system and supports the x86 and SPARCarchitectures. Each Solaris Container has its own host name, vNICor pNIC, and independent storage space.
• [2] Host overlay solution Figure 11.4 shows an example of the host overlay solution architecture in which PMs are connected to the fabric overlay network. In