A Network Artist : Do You Really Need VxLAN in a Modern Data Center ?

Thursday, April 23, 2020

Do You Really Need VxLAN in a Modern Data Center ?

It started as a thread on Linkedin yesterday when someone posted a sort of glimpse of his free Cisco ACI training on linkedin. Later a Network Architect joined the conversation thread by asking the Original poster "Would you prefer ACI or BGP EVPN VxLAN based standard fabric ?". Of course the Architect suggested he would prefer BGP EVPN VxLAN fabric (That's mouthful man :) )

As usual out of my curiosity (without being a DCN expert - Yes I only worked on 3 ACI projects and Zero Cisco Nexus 9K BGP EVPN VxLAN fabric (VTS) so far), I asked this self proclaimed Network Architect another question...very basic in my view:

In modern Data Center context, Where in VxLAN actually plays an important role. Or is it just another snowflake in Networking realm in order to keep your App guys happy since you are not good at Saying " NO " and push them to build better Apps those works well on L3 vs. Sure guys, I can help you with L2 stretches for your App Clusters and for those totally broken old legacy Apps which only works on L2 best.

The point being:

Most modern App works well on L3 in conjunction with DNS
From Cloud Native Apps perspective, L2 shouldn't happen to begin with
vMotion works well on L3 since 6.x release
Inter DC Cluster using L2 are well known bad design practices (Some people tried it with even firewalls and failed miserably)
At least in my personal view, Most Enterprises still run MLAG or vPC thingy for server redundancy compare to perfect world (Original Hyperscaler ideas) where you either run Routing with Servers and VxLAN boundary starts from Hypervisor layer to avoid broken MLAG and STP implementations
Let's agree on L3 based transport is much better compare to L2
Perhaps we have seen enough DC meltdowns in past due to broken L2 and broadcast loops by now ?
Did you ever notice that most Hyperscalers DCs don't run VxLAN in their DC fabrics (Pick your favorite one) - https://engineering.linkedin.com/blog/2016/03/project-altair--the-evolution-of-linkedins-data-center-network
Perhaps there are good reasons someone like AWS and AZURE don't let you run VxLAN between your On Prem DC and Virtual DC they host (VMware on AWS is a corner case and rather a joint GTM thingy)
How many of your Use Case requirements are persistent vs. temp in nature ( Migration and Maintenance were always temp requirements in my view at least)
What value Bridging over L3 really has today to begin with
Shouldn't we rather focus on solving real problems instead of pretending
Don't forget you can't have unlimited VxLANs in real world like most vendor slides tell you otherwise
Focus on your failure domains/blast radius and create a strategy to mitigate that - Which doesn't seems easy in VxLAN environment and more so in Controller based Clos Fabric (Single Plainer) running VxLAN
And since Overlays such as VxLAN seems to be the new norm in Enterprise DCs, Make sure you are familiar with some of well known Overlay Problems besides challenges such as Visibility, Security and so forth
Make sure you fully understand the VxLAN convergence process with EVPN added while understanding your dependencies on Physical Topology, Underlay Convergence and other nuances
Make sure you understand Leaky Abstractions & Grey Failures very well in context of VxLAN deployment
Understand some myths about Clos Fabric that your favorite vendor otherwise won't tell you - Part 1, Part 2, Part 3 & Fact Check

And of course I am totally ignoring:

Level of complexity VxLAN and EVPN brings into Design...You either got to read through few books to become an SME or your must be Genius
NSX over ACI never took off ... sort of
Most VxLAN implementations are still vendor specific and ACI runs modified VxLAN header
VxLAN & EVPN across multi vendors in Single environment... Good luck (Though things have improved a bit over time)
MPLS & SR in DCN Fabric...Perhaps needs a separate discussion
I am not aware of any strong one to one mapping in context of Overlay and Underlay correlation for simplified Day 2 and operations
VxLAN vs Geneve (What NSX-T uses now) vs STT vs NvGRE (Assuming you love MS) vs Dove....Needs separate discussion
This is not a scalability discussion but more around to find real world use cases for VxLAN
MS Fabric today (Not Originally) does run VxLAN since 2015. Spoke to few MS friends, seems like Bing was Broken from APP designs perspective so they had to do it in order to Keep Business and App guys happy....BTW who runs Bing anyway :)
If you are running HW VTEP with MLAG for servers. Perhaps you got to relook into your design and define what purpose VxLAN really serves
We still got plenty of other HW and SW in DC context that don't understand VxLAN and more importantly have no HW acceleration
Role of VxLAN offload on NIC, DPDK, SR-IOV and other stuff....perhaps needs a separate discussion
HW and SW VTEP Integration and how broken those are...let's keep it for another day
IPv6 DC/IPv6 Only DC and VxLAN v6... Perhaps need another thread
Control Plane, Data Plane & Policy Plane Security in Context of VxLAN - Requires a separate discussion
VxLAN Control Plane, Data Plane & Policy Plane in Multi DC, Multi Cloud and Active-Active DC - Needs separate discussion
Control Plane & Data Plane Optimization in VxLAN Context - Broadcast Suppression, Conversational MAC address Learning, BUM traffic (Watch out for those details in given implementation

So question remains - What value really VxLAN brings in the context of modern DC. (Please don't tell me scale, let's keep it as a topic of debate for another day) ?