Clos Fabrics AKA Spine & Leaf Architecture

Let's start the series with discussion of CLOS Fabrics AKA Spine & Leaf Architectures.

Now CLOS design is not fundamentally new, but most of the Network Engineers were not talking about it till recent times (Well...this is true to an extent). So as Network Engineer should you really care ?

Well you should start by asking why CLOS in first place ?

The major problem that CLOS fabric solves is about solving scalability issues. While scalability is a matter of context, it's not necessary that everyone needs or to be precise going too far about it.

Also CLOS fabric also doesn't define your Layer2 - Layer 3 boundaries itself. So you are pretty much dependent upon what works best for you from vendor implementation perspective while keeping your overall goal in mind. Now in theory Layer 3 Fabrics scale much better than Layer 2 Fabric. Here are some questions/Things you figure out about CLOS if you decide to go for it :

- What is the scale that you got to deal with ?
- What are technical and business requirements ?
- Your DC traffic is mostly east-west or north-south ?
- How you can minimize the state of the Core (Spine) to minimum ?
- How flooding works in your fabric ?
- How multicast is handled in fabric ?
- Where to define Layer2-Layer 3 boundary ?
- Your network is going to multi vendor now/In future ?
- How you gonna manage and monitor such large network ?
- How you gonna introduce security & Services such as Load Balancer ?
- How you gonna connect to external world ? (Border Spine Vs. Border Leaf) 
- Define you convergence requirements 
- You gonna need single or multi stage CLOS ?
- Your over subscription ratio ? (Usually 3:1 is good for most part)
- Understand your failure domains and impact they may have
- Do you need Spine to Spine or Leaf to Leaf connections to mitigate some of     
   failure scenario ?
- If you are going with Layer 3 fabric, is it going to be good idea to use 
   summarization ?
- EBGP vs IBGP (Also RR placement) in Layer 3 fabric ?


Even as an example, Cisco's famous buzzword these days ACI (Application Centric Infrastructure ) also uses Spine & Leaf design. It uses BGP EVPN (Some secret souce but soon EVPN will be there too) control plane and on top of which it uses VXLAN as Data Plane. So between Spine & Leaf (Single Stage) it uses Layer 3 fabric. The entire fabric is managed with a centralized command and control system called Cisco APIC Controller. With ACI you can go as far as 6 Spines at the moment and all services (e.g. load balancer), firewalls, external connectivity gets terminated on Leaf switches. For server redundancy (Bare Metal Or Virtual ) it uses our old friend Virtual Port Channel (vPC) but this time doesn't require directly connected interfaces among leaf switches for peer link and peer keep alive link functions. 

Cisco ACI is kind of build around another buzz word that you hear more often these days called SDN (Software Defined Networks). Now whether it fits into true SDN definition or not needs another discussion :).

In the mean while below is the list of URLs which you may find very handy to get started with CLOS:

http://packetpushers.net/podcast/podcasts/datanauts-011-understanding-leaf-spine-networks/

https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network

http://www.networkworld.com/article/2226122/cisco-subnet/clos-networks--what-s-old-is-new-again.html

http://searchdatacenter.techtarget.com/feature/The-case-for-a-leaf-spine-data-center-topology

http://searchdatacenter.techtarget.com/answer/Whats-the-best-data-center-network-topology

http://searchdatacenter.techtarget.com/feature/Data-center-network-design-moves-from-tree-to-leaf

http://www.excitingip.com/4490/distributed-coreleaf-spine-network-architecture-an-intro/

http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4448982&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F90%2F4359146%2F04448982.pdf%3Farnumber%3D4448982

http://etherealmind.com/which-network-topology/

http://packetpushers.net/network-topologies/

http://blog.ipspace.net/2014/04/security-in-leaf-and-spine-fabrics.html

http://www.thenetworkingdom.net/bgp-clos-networks/

http://thenetworksurgeon.com/cisco-spine-and-leaf-architecture-discussion-nexus-5500-vs-6001/

http://blog.ipspace.net/2012/04/full-mesh-is-worst-possible-fabric.html

http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p183.pdf

http://conferences.sigcomm.org/co-next/2013/program/p49.pdf

https://www.ietf.org/id/draft-ietf-rtgwg-bgp-routing-large-dc-09.txt

https://www.nanog.org/meetings/nanog55/presentations/.../Lapukhov.pdf

http://content.blubrry.com/packetpushers/Show_164-Cool_or_Hot-Lapukhov_Nkposong_s_BGP_SDN.mp3

http://www.juniper.net/us/en/local/pdf/whitepapers/2000565-en.pdf

https://cumulusnetworks.com/blog/routed-vmotion-why/

https://www.youtube.com/watch?v=nPKLe0M5yJU

https://www.youtube.com/watch?v=O8wU1qNlsyI

https://www.youtube.com/watch?v=fQ2l_v_53qQ

HTH...
Deepak Arora
Evil CCIE