Combining Etherchannel With Router On A Stick - When Two Evils Meet

It all started of as a thought in mind to write a new blog post on topic which I personally feel that no one talked about earlier or even any CCIE Lab Workbooks covered. But after some time my situation was like:

Okay, I'll talk about that later. First lets get started with what we are going to learn new today. The idea in mind was to combine two Evils together. 

Usually CCNA, CCNP & Evil CCIEs are familiar with these topics individually:

1. Router On A Stick
2. Ether-Channel

But what I thought was to combine these two together and come up with a new solution as -  Router On A Stick With Ether-Channel, Which will look something like :-)

Now after enough fun let's get into details:

 Physical Topology

Logical Topology

Now before we move ahead, for those which are not familiar with these two topics (Ether-Channel & Router On A Stick), I would recommend following Pre readings :

http://www.thebryantadvantage.com/CCNA640802CCENTRouterOnAStick.htm


http://www.thebryantadvantage.com/CCNA640802CCENTRouterOnAStickConfiguration.htm

http://www.thebryantadvantage.com/CCNA640802CCENTRouterOnAStickTroubleshooting.htm

http://blog.ipexpert.com/2009/12/23/explaining-etherchannel/

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_white_paper09186a0080092944.shtml

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml


Now lets move to Cli where I am going to show everything into 2 Important Steps:

1. Configuration
2. Verification


BTW,  I used a Cisco 3725 Router with 16 Port Switching Module to emulate the Switch part here so some of verification and configuration commands are specific to this setup as here Router is acting as a Switch.


SWITCH
======


SW#vlan database
SW(vlan)#vlan 10
VLAN 10 added:
    Name: VLAN0010
SW(vlan)#vlan 20
VLAN 20 added:
    Name: VLAN0020
SW(vlan)#apply
APPLY completed.
SW(vlan)#exit
APPLY completed.
Exiting....


SW#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
SW(config)#int f1/3
SW(config-if)#switchport mode access
SW(config-if)#switchport access vlan 10
SW(config-if)#spanning-tree portfast < Optional
%Warning: portfast should only be enabled on ports connected to a single host.
 Connecting hubs, concentrators, switches,  bridges, etc.to this interface
 when portfast is enabled, can cause temporary spanning tree loops.
 Use with CAUTION

%Portfast has been configured on FastEthernet1/3 but will only
 have effect when the interface is in a non-trunking mode.
SW(config-if)#exit

SW(config)#int f1/4
SW(config-if)#switchport mode access
SW(config-if)#switchport access vlan 20
SW(config-if)#spanning-tree portfast < Optional
%Warning: portfast should only be enabled on ports connected to a single host.
 Connecting hubs, concentrators, switches,  bridges, etc.to this interface
 when portfast is enabled, can cause temporary spanning tree loops.
 Use with CAUTION

%Portfast has been configured on FastEthernet1/3 but will only
 have effect when the interface is in a non-trunking mode.
SW(config-if)#exit


SW(config)#interface range f1/0 - 1
SW(config-if-range)#channel-group 1 mode on
SW(config-if-range)#exit
SW(config)#

SW(config)#interface port-channel 1 < All Trunk Config Goes To Port-Channel
SW(config-if)#switchport trunk encapsulation dot1q
SW(config-if)#switchport mode trunk
SW(config-if)#exit


R1
==

R1(config)#int port-channel 1 < Lets first Create Port-Channel Interface
R1(config-if)#exit


R1(config)#interface range f0/0 - 1
R1(config-if-range)#channel-group 1
%Interface MTU set to channel-group MTU 1500.

%Interface MTU set to channel-group MTU 1500.

R1(config-if-range)#no shut
R1(config-if-range)#exit


R1(config)#int port-channel 1.10 < Create Port-Channel Sub-interface for VLAN 10
R1(config-subif)#encapsulation dot1Q 10
R1(config-subif)#ip add 10.0.0.254 255.255.255.0
R1(config-subif)#no shut
R1(config-subif)#exit


R1(config)#int port-channel 1.20 < Create Port-Channel Sub-interface for VLAN 20
R1(config-subif)#encapsulation dot1Q 20
R1(config-subif)#ip add 20.0.0.254 255.255.255.0
R1(config-subif)#no shut
R1(config-subif)#end


Verification:
========

R1#sh ip int brief | exclude down
Interface              IP-Address      OK? Method Status                Protocol
FastEthernet0/0        unassigned      YES unset  up                    up
FastEthernet0/1        unassigned      YES unset  up                    up
Port-channel1          unassigned      YES unset  up                    up
Port-channel1.10       10.0.0.254      YES manual up                    up
Port-channel1.20       20.0.0.254      YES manual up                    up



SW#sh vlan-switch brief | include _10_|_20_
10   VLAN0010                         active    Fa1/3
20   VLAN0020                         active    Fa1/4



SW#sh interface trunk

Port      Mode         Encapsulation  Status        Native vlan
Po1       on           802.1q         trunking      1

Port      Vlans allowed on trunk
Po1       1-1005

Port      Vlans allowed and active in management domain
Po1       1,10,20

Port      Vlans in spanning tree forwarding state and not pruned
Po1       1,10,20



SW#sh spanning-tree vlan 10 | include forwarding
 Port 44 (FastEthernet1/3) of VLAN10 is forwarding
   Number of transitions to forwarding state: 1
 Port 321 (Port-channel1) of VLAN10 is forwarding
   Number of transitions to forwarding state: 1
SW#
SW#sh spanning-tree vlan 20 | include forwarding
 Port 45 (FastEthernet1/4) of VLAN20 is forwarding
   Number of transitions to forwarding state: 1
 Port 321 (Port-channel1) of VLAN20 is forwarding
   Number of transitions to forwarding state: 1


SW#sh etherchannel summary | b Group
Group Port-channel  Ports
-----+------------+-----------------------------------------------------------
1     Po1(SU)     Fa1/0(P)   Fa1/1(P)



HOST-A#sh ip int b | exclude unassignd|down
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            10.0.0.3        YES manual up                    up


HOST-A#ping 20.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 20.0.0.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/34/68 ms



HOST-B#sh ip int brief | e una|do
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            20.0.0.4        YES manual up                    up


HOST-B#ping 10.0.0.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/30/72 ms


R1#sh ip arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.0.0.3                2   c416.1064.0000  ARPA   Port-channel1.10
Internet  10.0.0.254              -   ca14.1c58.0008  ARPA   Port-channel1.10
Internet  20.0.0.4                2   c417.1064.0000  ARPA   Port-channel1.20
Internet  20.0.0.254              -   ca14.1c58.0008  ARPA   Port-channel1.20


R1#sh ip route | b ^G
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.0.0.0/24 is directly connected, Port-channel1.10
L        10.0.0.254/32 is directly connected, Port-channel1.10
      20.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        20.0.0.0/24 is directly connected, Port-channel1.20
L        20.0.0.254/32 is directly connected, Port-channel1.20



SW#sh mac-address-table dynamic
Non-static Address Table:
Destination Address  Address Type  VLAN  Destination Port
-------------------  ------------  ----  --------------------
ca14.1c58.0008          Dynamic       1     Po1
c416.1064.0000          Dynamic      10     FastEthernet1/3
c417.1064.0000          Dynamic      20     FastEthernet1/4


HOST-A#sh int f0/0 | i bia
  Hardware is Gt96k FE, address is c416.1064.0000 (bia c416.1064.0000)


HOST-B#sh int f0/0 | i bia
  Hardware is Gt96k FE, address is c417.1064.0000 (bia c417.1064.0000)


R1#sh int f0/0 | i bia
  Hardware is i82543 (Livengood), address is ca14.1c58.0008 (bia ca14.1c58.0008)

R1#sh int f0/1 | i bia
  Hardware is i82543 (Livengood), address is ca14.1c58.0008 (bia ca14.1c58.0006)

R1#sh int Po1 | i bia
  Hardware is FEChannel, address is ca14.1c58.0008 (bia 0000.0000.0000)





And of course that thing which made smashing my head against the key-board, Here are the details:

https://learningnetwork.cisco.com/thread/50039?tstart=0


Further Readings:


http://www.cisco.com/en/US/prod/collateral/routers/ps5855/prod_white_paper0900aecd806f698a.pdf

HTH...
Deepak Arora
Evil CCIE

HSRP Preemption On Standby Router - Required Or Not ?

Often I ask this question to CCNPs during interviews - "Do we require PREEMPT command on HSRP Standby router or not ? "

And often candidates either reply "NO" or looks Struggling.

Although Short answer is "YES". But here is the detailed explanation using a test bed.

For those which are new to HSRP. I would recommend first read following posts from Gr8 CCNP Intructor Chris Bryant, CCIE#12933


http://www.thebryantadvantage.com/CCNPCertificationBCMSNExamHSRPInterfaceTracking.htm

http://www.thebryantadvantage.com/CCNPCertificationExamBCMSNHSRPPriorityActive.htm

http://www.thebryantadvantage.com/CCNPCertificationBCMSNExamHSRPMACAddress.htm


Now lets get started with topology details later followed by Detailed Explanation using Test Bed:

 

 

 

 

 

 

 

Further Readings:

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_q_and_a_item09186a00800a9679.shtml

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a0080094a91.shtml

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a0080094afd.shtml 

 

HTH...

Deepak Arora

EVIL CCIE

Riding A 6500

Spanning Tree for Dummies - Poem From Radia Perlman

“Algorhyme”

I think that I shall never see

a graph more lovely than a tree.

A tree whose crucial property

is loop-free connectivity.

A tree that must be sure to span

so packets can reach every LAN.

First, the root must be selected.

By ID, it is elected.

Least-cost paths from root are traced.

In the tree, these paths are placed.

A mesh is made by folks like me,

then bridges find a spanning tree.

 

 

Further Readings: http://www.intelfreepress.com/news/the-many-sides-of-radia-perlman/

 

HTH...

Deepak Arora

Evil CCIE

My Music Zone

IPv6 Deep Dive Training

This whole week I'll be going through IPv6 Advanced Training from Authorized Cisco Partner. The first day was more focussed on fundamentals and all which I have already went through during my CCIE R&S. But still picked up few new and nice things. Also R&S Lab was more focussed on Routing & Tunneling Techniques. Where this training is also looking at other aspects like WINDOWS/LINUX/MAC side configurations which I never tried much earlier. Also other important deev dives on topics like DHCPv6, DNSv6, v6 PE-CE & v6 mobility etc. Following is the lab topology that each candidate have access to. I'll post more details next week.

Class Outline:

http://tools.cisco.com/GlobalLearningLocator/courseDetails.do?actionType=executeCourseDetail&courseID=5653

HTH...
Deepak Arora
Evil CCIE

Address Resolution Protocol - ARP

Often I have seen CCNA Candidates struggling with understanding ARP & Proxy ARP. So I always recommend such guys two things :

1. Read through the following document:

http://www.cisco.com/en/US/docs/ios/ipaddr/configuration/guide/iad_cfg_arp_ps6350_TSD_Products_Configuration_Guide_Chapter.html

2. Lab up ARP and Inverse ARP on real equipments and verify the operation using debugs.


HTH...
Deepak Arora
Evil CCIE

Network Design Challenge

PHYSICAL TOPOLOGY

       LOGICAL TOPOLOGY

Recently one of the customer I am working for came up with an interesting requirement.

Current customer setup look alike something similar to topology (logical) given above. Customer has two locations per say connected through back to back T1 Circuits. In current setup R1-R3 is the primary path, and if a failure occurs in that case traffic should fall back to backup path using R2-R4. Customer is using One HSRP group on each side for auto failover purpose and currently static routing is in place.
 
So considering all this, following is how over all traffic flow looks like at the moment from A to C and from B to C perspective:
 

Hmmm... sounds like a pretty flat and easy network... isn't it ? :-)


Now here is the new requirement: As you guys can see we are wasting our bandwidth here on the backup circuit between R2-R4. And Bandwidth in our real life means $$$$$$ :-)

So here is new desired flow: When Host A talks to Host C, It should use Path through R1-R3. At the same time if Host B talks to Host C, It should use Path through R2-R4. At the same time both links should be redundant for each other for all traffic flow. So based on this requirement , the traffic flow needs to look like following:

As you can see , we need to account 8 potential FAILURE DOMAINS to ensure our fail-over scenarios works just fine. 

Also requirement is to ensure we don't have any asymmetric routing into the network. And of course NO dynamic routing protocol should be introduced.

So how you gonna do this ? :-) , since from very high level perspective it looks very easy. But it ain't :-)

            
Current Configuration
++++++++++++++                                                                                                                        
SW-A
=====
!
en
!
conf t
!
ho SW-A
!
no ip do lo
!
no cdp run
!
ip routing
!
int f1/1
 sw mo acc
 sw acc vl 10
 exit
!
int vlan 10
 ip add 10.0.0.1 255.255.255.0
 no sh
 exit
!
ip route 20.0.0.0 255.255.255.0 12.0.0.1
ip route 30.0.0.0 255.255.255.0 12.0.0.1
!
int f1/0
 no switchport
 ip add 12.0.0.2 255.255.255.0
 no sh
 exit
!
end

!


SW-B
=====
!
en
!
conf t
!
ho SW-B
!
no ip do lo
!
no cdp run
!
ip routing
!
int f1/0
 no switchport
 ip add 34.0.0.2 255.255.255.0
 no sh
 exit
!
int f1/1
 sw mode access
 sw acc vl 20
 exit
!
int f1/2
 sw mode access
 sw acc vl 30
 exit
!
int vlan 20
 ip add 20.0.0.1 255.255.255.0
 no sh
 exit
!
int vlan 30
 ip add 30.0.0.1 255.255.255.0
 no sh
 exit
!
ip route 0.0.0.0 0.0.0.0 34.0.0.1
!
end
!

R1
====
!
en
!
conf t
!
ho R1
!
no ip do lo
!
int f0/0
 ip add 12.0.0.111 255.255.255.0
 standby 1 ip 12.0.0.1
 standby 1 pri 105
 standby 1 pree
 standby 1 track s1/0
 exit
!
int s1/0
 ip add 13.0.0.1 255.255.255.0
 encap ppp
 no sh
 exit
!
ip route 20.0.0.0 255.255.255.0 13.0.0.3
ip route 30.0.0.0 255.255.255.0 13.0.0.3
ip route 0.0.0.0 0.0.0.0 13.0.0.3
!
ip route 10.0.0.0 255.255.255.0 12.0.0.2
!
end
!


R2
===
!
en
!
conf t
!
ho R2
!
no ip do lo
!
no cdp run
!
int f0/0
 ip add 12.0.0.112 255.255.255.0
 standby 1 ip 12.0.0.1
 standby 1 pree
 no sh
 exit
!
int s1/0
 ip add 24.0.0.2 255.255.255.0
 encap ppp
 no sh
 exit
!
ip route 20.0.0.0 255.255.255.0 24.0.0.4
ip route 30.0.0.0 255.255.255.0 24.0.0.4
ip route 0.0.0.0 0.0.0.0 24.0.0.4
!
ip route 10.0.0.0 255.255.255.0 12.0.0.2
!
end
!

R3
===
!
en
!
conf t
!
ho R3
!
no ip do lo
!
no cdp run
!
int f0/0
 ip add 34.0.0.3 255.255.255.0
 standby 1 ip 34.0.0.1
 standby 1 pri 105
 standby 1 pree
 standby 1 track s1/0
 no sh
 exit
!
int s1/0
 ip add 13.0.0.3 255.255.255.0
 encap ppp
 no sh
 exit
!
ip route 10.0.0.0 255.255.255.0 13.0.0.1
ip route 0.0.0.0 0.0.0.0 13.0.0.1
!
ip route 20.0.0.0 255.255.255.0 34.0.0.2
ip route 30.0.0.0 255.255.255.0 34.0.0.2
!
end
!
 

R4
===
!
en
!
conf t
!
ho R4
!
no ip do lo
!
no cdp run
!
int f0/0
 ip add 34.0.0.4 255.255.255.0
 standby 1 ip 34.0.0.1
 standby 1 pree
 no sh
 exit
!
int s1/0
 encap ppp
 ip add 24.0.0.4 255.255.255.0
 no sh
 exit
!
ip route 10.0.0.0 255.255.255.0 24.0.0.2
ip route 0.0.0.0 0.0.0.0 24.0.0.2
!
ip route 20.0.0.0 255.255.255.0 34.0.0.2
ip route 30.0.0.0 255.255.255.0 34.0.0.2
!
end
!

So let's see who among you is a Real Network Warrior  :-)

Final Config

R1
====

!
en
!
conf t
!
ho R1
!
no ip do lo
!
ip sla 2
 icmp-echo 34.0.0.3 source-interface f0/0
 freq 5
 timeout 5000
 exit
!
ip sla schedule 2 life forever start-time now
!
track 2 rtr 2 reachability
!
int f0/0
 ip add 12.0.0.111 255.255.255.0
 standby 1 ip 12.0.0.1
 standby 1 pri 105
 standby 1 pree
 standby 1 track s1/0
 standby 1 track 2
 standby 2 ip 12.0.0.11
 standby 2 pree
 no sh
 exit
!
int s1/0
 ip add 13.0.0.1 255.255.255.0
 encap ppp
 no sh
 exit
!
ip route 20.0.0.0 255.255.255.0 13.0.0.3
ip route 0.0.0.0 0.0.0.0 13.0.0.3
!
ip route 10.0.0.0 255.255.255.0 12.0.0.2
!
end
!





R2
===

!
en
!
conf t
!
ho R2
!
no ip do lo
!
no cdp run
!
ip sla 1
 icmp-echo 34.0.0.4 source-interface s1/0
 time 500
 freq 3
 exit
!
ip sla schedule 1 life forever start-time now
!
track 2 rtr 1 reachability
!
int f0/0
 ip add 12.0.0.112 255.255.255.0
 standby 1 ip 12.0.0.1
 standby 1 pree
 standby 2 ip 12.0.0.11
 standby 2 pri 105
 standby 2 pree
 standby 2 track 2
 no sh
 exit
!
int s1/0
 ip add 24.0.0.2 255.255.255.0
 encap ppp
 no sh
 exit
!
ip route 30.0.0.0 255.255.255.0 24.0.0.4
ip route 0.0.0.0 0.0.0.0 24.0.0.4
!
ip route 10.0.0.0 255.255.255.0 12.0.0.2
!
end
!


SW-A
=====

!
en
!
conf t
!
ho SW-A
!
no ip do lo
!
no cdp run
!
ip routing
!
int f1/1
 sw mo acc
 sw acc vl 10
 exit
!
int vlan 10
 ip add 10.0.0.1 255.255.255.0
 no sh
 exit
!
ip route 20.0.0.0 255.255.255.0 12.0.0.1
ip route 30.0.0.0 255.255.255.0 12.0.0.11
!
int f1/0
 no switchport
 ip add 12.0.0.2 255.255.255.0
 no sh
 exit
!
end
!



R3
===


!
en
!
conf t
!
ho R3
!
no ip do lo
!
no cdp run
!
int f0/0
 ip add 34.0.0.3 255.255.255.0
 standby 1 ip 34.0.0.1
 standby 1 pri 105
 standby 1 pree
 standby 1 track s1/0
 no sh
 exit
!
int s1/0
 ip add 13.0.0.3 255.255.255.0
 encap ppp
 no sh
 exit
!
ip route 10.0.0.0 255.255.255.0 13.0.0.1
ip route 0.0.0.0 0.0.0.0 13.0.0.1
!
ip route 20.0.0.0 255.255.255.0 34.0.0.2
ip route 30.0.0.0 255.255.255.0 34.0.0.2
!
ip route 24.0.0.0 255.255.255.0 34.0.0.4
!
access-l 100 permit ip 30.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
!
ip sla 1
 icmp-echo 24.0.0.4 source-ip 34.0.0.3
 timeout 5000
 frequency 3
!
ip sla schedule 1 life forever start-time now
!
ip sla 3
 icmp-echo 12.0.0.112 source-interface FastEthernet0/0
 frequency 5
!
ip sla schedule 3 life forever start-time now
!
track 1 rtr 1 reachability
!
track 3 rtr 3 reachability
!
track 2 list boolean and
 object 1
 object 3
!
route-map PBR per 10
 match ip address 100
 set ip next-hop verify-availability 34.0.0.4 1 track 2
route-map PBR per 20
 exit
!
int f0/0
 ip policy route-map PBR
 no ip route-cache cef
 exit
!
ip sla 5
 icmp-echo 12.0.0.111 source-interface Serial1/0
 frequency 5
 timeoute 5000
!
ip sla schedule 5 life forever start-time now
!
track 5 rtr 5 reachability
!
int f0/0
 standby 1 track 5
 exit
!
end
!



R4
===

!
en
!
conf t
!
ho R4
!
no ip do lo
!
no cdp run
!
int f0/0
 ip add 34.0.0.4 255.255.255.0
 standby 1 ip 34.0.0.1
 standby 1 pree
 no sh
 exit
!
int s1/0
 encap ppp
 ip add 24.0.0.4 255.255.255.0
 no sh
 exit
!
ip route 10.0.0.0 255.255.255.0 24.0.0.2
ip route 0.0.0.0 0.0.0.0 24.0.0.2
!
ip route 20.0.0.0 255.255.255.0 34.0.0.2
ip route 30.0.0.0 255.255.255.0 34.0.0.2
!
end
!





SW-B
=====


!
en
!
conf t
!
ho SW-B
!
no ip do lo
!
no cdp run
!
ip routing
!
int f1/0
 no switchport
 ip add 34.0.0.2 255.255.255.0
 no sh
 exit
!
int f1/1
 sw mode access
 sw acc vl 20
 exit
!
int f1/2
 sw mode access
 sw acc vl 30
 exit
!
int vlan 20
 ip add 20.0.0.1 255.255.255.0
 no sh
 exit
!
int vlan 30
 ip add 30.0.0.1 255.255.255.0
 no sh
 exit
!
ip route 0.0.0.0 0.0.0.0 34.0.0.1
!
end
!

HTH...
DEEPAK ARORA
Evil CCIE

Cisco Fabric Extender AKA FEX - Nexus 2000 Series

Cisco NEXUS 2000 Series Fabric Extender (FEX)

In today’s post, we are going to dive into new Concept – Fabric Extender. If you are familiar with Traditional Modular Switches AKA Distributed Switching Architecture such as Catalyst 6500/6500 E series or 4500/4500 X series than Fabric Extender is going to be a piece of cake for you. In Distributed Switching Architecture we basically have following components:

Ø  Supervisor Engine – A supervisor engine is basically Mind of switch responsible for management & control plane management.

Ø  Line Card/ Modules - The End Hosts or other devices gets connected on Line Cards/Modules, the basic function of these is to take care of Data Plane/ Forwarding Plane.

Ø  Back Plane/ X-Bar Fabric - The Supervisor Engine/ Engines talk to line cards & Modules using High speed backplane circuitry.

Now in older Distributed Switching platforms all these components reside in a single Chassis. What Cisco did to take this concept a step further is came up with something known as Fabric Extender. A fabric extender basically comes as Cisco Nexus 2000 Series. A fabric extender works as a line card of its parent switch which could be either Nexus 7k or Nexus 5k.

The communication between Parent Switch and Fabric Extender happens through something called Fabric Ports. Let’s see how our new Design with Parent Switch and Fabric Extender Looks like:

So as you can see, the Fabric Extender (FEX) is now the device (Remote Line Cards) to which end hosts gets attached. The FEX further gets connected to Parent Switch which is essentially acting as supervisor engine. It’s the Supervisor Engine where entire configuration is saved which means FEX don’t save any of the configurations locally. Also it’s the parent switch where all forwarding decisions are made. Which means if HOST A on FEX 1 wants to talk to another HOST B connected on same FEX 1, the traffic has to go to parent Switch and sent back towards the FEX as shown in diagram below:

Which is obviously not an idle scenario though but this is the way FEX has been designed. But on the other hand introduction of FEX offers us couple of benefits as well:

   Ø  Reduce Cable Runs

   Ø  Reduces Management Point ( Since Parent Switch Controls all FEXs)

   Ø  Reduces IOS Management & Standardization Load (The FEX always Runs same NX-OS as Parent SW, FEX gets shipped with No NX-OS Image. When we connect FEX to Parent switch and provision it, it downloads the NX-OS from parent switch and uses it)

   Ø  Enables Parent Switch (7K/5K) to become a high density access layer switch

   Ø  STP Free Access Layer ( The fabric links between Parent Switch & FEX runs no STP, on the flip side FEX Host Interfaces(HIF) cannot be used to connect any further switches or devices running Spanning-Tree. BPDU Guard feature is by default enabled on HIF and cannot be disabled. Which means as soon as you plug any device to the FEX running STP, the port will get error disabled)

   Ø  QOS & Security Management from Parent Switch

   Ø  All Troubleshooting From Parent Switch
  Ø Less number of OOB connection

    Since in most of the real world designs you would want to Pair your FEX with Nexus 5K, so we will be discussing basic FEX configuration from Nexus 5K perspective:

   The official NX-OS version in CCIE DC lab for Nexus 5548 is 5.1(3) & FEX models available in lab will be 2232 & 2224.

   Let’s review the topology and relative configuration now:

  Step 1 > Enable FEX feature set on Nexus 5k :

                       5k(config)# feature fex

  Step 2 > Create a Port Channel :

                       5k(config)# interface port-channel 101

  Step 3 > Configure Port Channel to Act as Fabric Link:

                       5k(config-if)# switchport mode fex-fabric

  Step 4 > Assign Associate ID to Remote FEX (From range 101 – 199)

                       5k(config-if)# fex associate 101

  Step 5 > Map Fabric Interfaces to Port Channel:

                       5k(config)# interface e1/1 - 4

                       5k(config-if-range)# channel-group 101

    Once this configuration will be done, we should be ready use FEX. The associate ID gets prepended to FEX interface for identification. For example if FEX port were like : E1/1, E1/2…., After association ID 101 assigned to FEX, on parent switch the ports will now appear as: E101/1/1, E101/1/2. Now any configuration you apply under these ports will be pushed to FEX itself.

   Couple of verification commands:

   Ø  sh fex

   Ø  sh fex detail

   Ø  sh interface status fex  = sh interface status (in regular IOS)

       Couple of limitations with FEX :
  
   > FEX Doesn't support Private VLANs
   > Any port available on FEX cannot be SPAN destination port
   > FEX Model like 2148 Can't be associated with Nexus 7K
   > So far only FEX model 2232PP supports FCOE on FEX ports 
   > On Nexus 7k, the non-default VDC gets access to FEX feature only if  
          feature is installed and enabled under default VDC
   >  In Nexus 7-K all the uplinks and host ports of a Fabric Extender 
            belong to a single VDC. The ports cannot be allocated or split 
            among multiple VDCs.
   > Nexus 7k F1 modules are though Layer 2 cards, but still doesn't support 
           FEX
   > You can configure the Fabric Extender host interfaces as edge 
          ports only.The interface is placed in an error disabled state if a   
          downstream switch is detected.
   > The Cisco Nexus 2148 Fabric Extender does not support frames 
            with the dot1p vlan 0 tag


   
  And here is most interesting one. Just take a look at Pic below.

   
  Actually you are looking at Back (rear) side of Nexus 2000 FEX, 
   and default Air-flow is from front to back.

Further Readings:


  7k Fex Command Ref:

    http://www.cisco.com/en/US/docs/switches/datacenter/sw/6_x/nx-os/fex/command/reference/n7k_fex_cmd.html

 5k Fex Command Ref:

   http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/command/reference/fex/n5k-fex-cr.html


   2k NX-OS Software Rel 6.1 Configuration Guide:

  http://www.cisco.com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_6_1/b_Cisco_Nexus_2000_Series_NX-OS_Fabric_Extender_Software_Configuration_Guide_Release_6.x.html

  
   2k Cisco Nexus 2000 NX-OS Software Rel 5.2 Configuration Guide:

  http://www.cisco.com/en/US/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_521/b_Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_5_2.html


    Cisco Nexus 5000 Series NX-OS Layer 2 Switching Configuration Guide, Release 5.2:

  http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/521_n1_1/b_Nexus_5000_Layer2_Config_521N11_chapter_010000.html

Nexus 2k Airflow:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/product_bulletin_c25-680197.html

HTH...
Deepak Arora
Evil CCIE