This post will describe one option to implement a VXLAN enabled fabric. I will use VSR router to create the fabric and create the overlay using VXLAN.
I already gave a general overview about VXLAN in an earlier post here:
VXLAN: What is VXLAN
VXLAN Implementation Introduction
After the layer 2 VPN feature is enabled, I need to create the virtual switching instance:
arp suppression enable
There many options for the VSI, which can be configured. I will only use three of them. The description, to insert more information for other admins, which maybe need to work and configure the device.
The most important option is the VXLAN option, which defines for which VXLAN the VSI is used. Every VXLAN needs one VSI.
The “arp suppression enable” one, will allow the VTEP to answer local ARP requests for remote devices, to reduce the amount of traffic in the VXLAN tunnel.
The next step is to create the VXLAN tunnel, which is used to send the encapsulated packets over the fabric. Different VXLAN networks can use the same VXLAN tunnel.
This tunnel can be created manually between VTEP’s, which is good for environments with a small number of VTEP’s or environments which are very static. For environments with more VTEP’s or which grow dynamically, a automatic mechanism is available to create the tunnels between different VTEP’s, My suggestion is to use the automatic mechanism to create the tunnels as this will make future changes much more easy.
The manual way is described in the manual, which is linked at the end of the post.
The automatic procedure is using the Enhanced Neighbor Discovery Protocol (ENDP). ENDP is used to discover all available VTEP’s in a network. Share the information among those VTEP’s about other VTEP’s to make sure that they establish a VXLAN tunnel among them and build a full mesh, between all VTEP’s.
The ENPD infrastructure consists of two parts. An ENDS (Enhanced Neighbor Discovery Server) which is used to collect all information about VTEP’s and share them with other VTEP’s. It is like a directory server.
The ENDC (Enhanced Neighbor Discovery Client) is a VTEP’s which connects to the ENDS to announce himself to ENDS and to get the information about other ENDC’s in the network, to create VXLAN tunnels to those ENDC’s.
The ENDS can be a VTEP or a another device, not involved in the VXLAN stuf. In my setup I use one of the Spine devices to be the ENDS.
To configure the ENDS the following configuration is required:
The source interface, used in this example is a previously created loopback interface and is distributed in the fabric using OSPF.
The network-id is used group different VXLAN domains and needs to be the same on the server and on the client.
The last command (line 4) enabled the server and makes the server accessible for the clients.
To configure the client, the following steps are required:
It looks the same as the server configuration. The only difference is the last line. Here, the client is enabled and the server address is configured.
If everything is working correctly the server should list all ENDC’s:
This command will show the used tunnel interface and the number of attached clients.
The show command above will list the connected clients. Make sure, you see all VTEP’s, which are running ENDP as clients.
To get the information if all clients are up and running, you can use the display command above.
As all VTEP’s are now aware of all the other VTEP’s, they should have established VXLAN tunnel among each other. On a VTEP, I will very this, using this command:
The above command will list all create tunnels and the detailed information for those tunnels. Leaf 1 has established tunnels with all other VTEP’s (Tunnel2, Tunnel3 and Tunnel4) and they are all up and running.
If a new VTEP is added and ENDP is enabled on this VTEP, the existing VTEP’s will create a tunnel to the new one as well.
The next step, is to assign VXLAN networks to VXLAN tunnels. VXLAN tunnels can have multiple VXLAN networks assigned to them and VXLAN networks can be assigned to multiple VXLAN tunnels.
This could be done manually, but I prefer to do this automatically. To assign VXLAN networks to VXLAN tunnels IS-IS is used. With IS-IS, the VTEP’s will exchange information about available VXLAN networks and if two VTEP’s have the same VXLAN network configured, they will put this one automatically on the VXLAN tunnel between them. This makes it easy, as I only have to create the VXLAN networks on the VTEP’s, without the need to configure all tunnels.
The first step, is to reserve one VXLAN networks for IS-IS. This is used to exchange the information and needs to be the same on all VTEP’s:
The last step is to enable IS-IS:
The second line, will enable IS-IS and allow the exchange about VXLAN networks. The third line is already the first optimization step. This will make sure, that learned local MAC addresses are distributed in the network, using IS-IS. This should reduce the amount of broadcast in the network, together with the previously mentioned arp suppression feature.
To assign real traffic to the VXLAN network, I need to tell the device, from which port, the incoming traffic should assigned to the VXLAN:
I imply assign the VSI to the interface, the host (in this case Server1) is connected to. Every frame, which arrives at this interface will now be switched using the information from the VSI, which takes all the endpoints behind the other VTEP’s into account. If the destination is behind a different VTEP, the frame is send through the VXLAN tunnel to the other VTEP.
To check if everything is working correctly I will use the following display commands:
The command above will list all VSI’s and the assigned tunnels. The VSI “site-1” is only configured on Leaf 1 and Leaf 3. The tunnel between those two VTEP’s is Tunnel3. The VXLAN id for this VXLAN network is 1. Everything looks correct on this one.
Let’s check if the MAC learning is working. Currently, Leaf 1 and Leaf 3 do not know any MAC address. I will start forcing Leaf 1 to learn the MAC address of Server 1 by pinging and unknown host (the host does not exists) in the networks. Leaf 1 will learn the MAC through the ARP request:
As Leaf 1 is now aware of the MAC address of Server 1, this MAC address is distributed to Leaf 3 by IS-IS:
The difference from Leaf 1 is, that the State is IS-IS and that there is a Tunnel assigned to the MAC, which is Tunnel2, as on Leaf 3, the tunnel to Leaf 1 is Tunnel2. If Server 3, which is connected to Leaf 3 is trying to reach Server 1, no broadcast is needed as the MAC address is already known to the VTEP and the VTEP will answer the ARP request, instead of broadcasting the request through the VXLAN tunnels.
At the end, I have created the following VXLAN assignments:
VXLAN Documentation:Those are the essential steps, to create a VXLAN environment. Of curse, there are many more options to tweak and enhance the functionality.
If you have any questions please use the comment function below. If you would like to provide feedback please contact me or use the comment function as well.