Cisco Fabric Extender AKA FEX - Nexus 2000 Series

Cisco NEXUS 2000 Series Fabric Extender (FEX)

In today’s post, we are going to dive into new Concept – Fabric Extender. If you are familiar with Traditional Modular Switches AKA Distributed Switching Architecture such as Catalyst 6500/6500 E series or 4500/4500 X series than Fabric Extender is going to be a piece of cake for you. In Distributed Switching Architecture we basically have following components:

Ø  Supervisor Engine – A supervisor engine is basically Mind of switch responsible for management & control plane management.

Ø  Line Card/ Modules - The End Hosts or other devices gets connected on Line Cards/Modules, the basic function of these is to take care of Data Plane/ Forwarding Plane.

Ø  Back Plane/ X-Bar Fabric - The Supervisor Engine/ Engines talk to line cards & Modules using High speed backplane circuitry.

Now in older Distributed Switching platforms all these components reside in a single Chassis. What Cisco did to take this concept a step further is came up with something known as Fabric Extender. A fabric extender basically comes as Cisco Nexus 2000 Series. A fabric extender works as a line card of its parent switch which could be either Nexus 7k or Nexus 5k.

The communication between Parent Switch and Fabric Extender happens through something called Fabric Ports. Let’s see how our new Design with Parent Switch and Fabric Extender Looks like:

So as you can see, the Fabric Extender (FEX) is now the device (Remote Line Cards) to which end hosts gets attached. The FEX further gets connected to Parent Switch which is essentially acting as supervisor engine. It’s the Supervisor Engine where entire configuration is saved which means FEX don’t save any of the configurations locally. Also it’s the parent switch where all forwarding decisions are made. Which means if HOST A on FEX 1 wants to talk to another HOST B connected on same FEX 1, the traffic has to go to parent Switch and sent back towards the FEX as shown in diagram below:

Which is obviously not an idle scenario though but this is the way FEX has been designed. But on the other hand introduction of FEX offers us couple of benefits as well:

   Ø  Reduce Cable Runs

   Ø  Reduces Management Point ( Since Parent Switch Controls all FEXs)

   Ø  Reduces IOS Management & Standardization Load (The FEX always Runs same NX-OS as Parent SW, FEX gets shipped with No NX-OS Image. When we connect FEX to Parent switch and provision it, it downloads the NX-OS from parent switch and uses it)

   Ø  Enables Parent Switch (7K/5K) to become a high density access layer switch

   Ø  STP Free Access Layer ( The fabric links between Parent Switch & FEX runs no STP, on the flip side FEX Host Interfaces(HIF) cannot be used to connect any further switches or devices running Spanning-Tree. BPDU Guard feature is by default enabled on HIF and cannot be disabled. Which means as soon as you plug any device to the FEX running STP, the port will get error disabled)

   Ø  QOS & Security Management from Parent Switch

   Ø  All Troubleshooting From Parent Switch
  Ø Less number of OOB connection

    Since in most of the real world designs you would want to Pair your FEX with Nexus 5K, so we will be discussing basic FEX configuration from Nexus 5K perspective:

   The official NX-OS version in CCIE DC lab for Nexus 5548 is 5.1(3) & FEX models available in lab will be 2232 & 2224.

   Let’s review the topology and relative configuration now:

  Step 1 > Enable FEX feature set on Nexus 5k :

                       5k(config)# feature fex

  Step 2 > Create a Port Channel :

                       5k(config)# interface port-channel 101

  Step 3 > Configure Port Channel to Act as Fabric Link:

                       5k(config-if)# switchport mode fex-fabric

  Step 4 > Assign Associate ID to Remote FEX (From range 101 – 199)

                       5k(config-if)# fex associate 101

  Step 5 > Map Fabric Interfaces to Port Channel:

                       5k(config)# interface e1/1 - 4

                       5k(config-if-range)# channel-group 101

    Once this configuration will be done, we should be ready use FEX. The associate ID gets prepended to FEX interface for identification. For example if FEX port were like : E1/1, E1/2…., After association ID 101 assigned to FEX, on parent switch the ports will now appear as: E101/1/1, E101/1/2. Now any configuration you apply under these ports will be pushed to FEX itself.

   Couple of verification commands:

   Ø  sh fex

   Ø  sh fex detail

   Ø  sh interface status fex  = sh interface status (in regular IOS)

       Couple of limitations with FEX :
  
   > FEX Doesn't support Private VLANs
   > Any port available on FEX cannot be SPAN destination port
   > FEX Model like 2148 Can't be associated with Nexus 7K
   > So far only FEX model 2232PP supports FCOE on FEX ports 
   > On Nexus 7k, the non-default VDC gets access to FEX feature only if  
          feature is installed and enabled under default VDC
   >  In Nexus 7-K all the uplinks and host ports of a Fabric Extender 
            belong to a single VDC. The ports cannot be allocated or split 
            among multiple VDCs.
   > Nexus 7k F1 modules are though Layer 2 cards, but still doesn't support 
           FEX
   > You can configure the Fabric Extender host interfaces as edge 
          ports only.The interface is placed in an error disabled state if a   
          downstream switch is detected.
   > The Cisco Nexus 2148 Fabric Extender does not support frames 
            with the dot1p vlan 0 tag


   
  And here is most interesting one. Just take a look at Pic below.

   
  Actually you are looking at Back (rear) side of Nexus 2000 FEX, 
   and default Air-flow is from front to back.

Further Readings:


  7k Fex Command Ref:

    http://www.cisco.com/en/US/docs/switches/datacenter/sw/6_x/nx-os/fex/command/reference/n7k_fex_cmd.html

 5k Fex Command Ref:

   http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/command/reference/fex/n5k-fex-cr.html


   2k NX-OS Software Rel 6.1 Configuration Guide:

  http://www.cisco.com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_6_1/b_Cisco_Nexus_2000_Series_NX-OS_Fabric_Extender_Software_Configuration_Guide_Release_6.x.html

  
   2k Cisco Nexus 2000 NX-OS Software Rel 5.2 Configuration Guide:

  http://www.cisco.com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/b_Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2.html


    Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide, Release 5.2:

  http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/521_n1_1/b_Nexus_5000_Layer2_Config_521N11_chapter_010000.html

Nexus 2k Airflow:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/product_bulletin_c25-680197.html

HTH...
Deepak Arora
Evil CCIE