Fibre Channel Over Ethernet
+++++++++++++++++++++++++++
- Unified Fabric = Unified Wire = Converged Ethernet = Data Center Ethernet = Data Center Bridging
- In unified fabric - Ethernet & Fibre channel both run on common infrastructure
- FCoE Initialization Protocol (FIP) is control plane protocol for FCoE running between Initiator & FCOE Switch/FCoE Forwarder(FCF)
- ENode in FCoE = N port in FC
- Virtual Fibre Channel (VFC) interface is logical interface mapped with physical interfaces. FIP Runs between ENode & VFC using P2P Adjacency
- Virtual Port Types in FCOE
> VN Port = N Port in FC
> VF Port = F Port in FC
> VE Port = E Port in FC < Used in Multi Hop FCOE
> VTE Port = TE Port in FC
- FCoE replaces layer 1 & layer 2 tranport for FC but all upper layer FC services remains same like Zoning, Domain IDs, FSPF, FLOGI, FCNS etc
- FIP is control place of FCoE and FCoE itself is the actual data plane
# FIP
++++++
- FIP has new Ether Type as 0x8914
- FIP is used to discover FCFs and perform FLOGI
# FCOE
+++++++
- New Ethertype 0x8906
- Max length of 2240 bytes which implies jumbo frames are required
www.t11.org/fcoe
# FCoE Addressing
+++++++++++++++++
- Fibre Channel uses 3 bytes FCIDs
- Ethernet uses 6 byte MAC Addresses
- FCoE ENode gets a Fabric Provided MAC Address (FPMA) for FCoE
- During FIP, ENode is assigned a 3 byte FCID
- FLOGI now is part of FIP
- FCF is configured with a 3 byte FCoE MAC Address Prefix (FC-MAP)
- ENode appends FC-MAP to FCID
- 3 byte FC-MAP + 3 byte FCID = FPMA
# Configuring FCOE
+++++++++++++++++++
- feature fcoe < Feature FCOE can only be enabled from Admin VDC in Nexus 7k
- feature lldp
- Create a VSAN
vsan database
vsan 1010
vsan 1010 interface fc1/10
- Associate VSAN to a VLAN
vlan 1010
fcoe vlan 1010
- Configure a VFC
- Associate Physical Ethernet to a VFC
int vfc117
bind interface e1/17
switchport trunk allowed vsan 1010
- Assign VFC to the VSAN
vsan database
vsan 1010 interface vfc117
- Configure physical ethernet as a trunk
int e1/17
shut
switchport mode trunk
spanning-tree port type edge trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 10 ,1010
- Activate Interfaces
int vfc117
no shut
int e1/17
no shut
sh int vfc117 < Look for trunking & VSAN should be up
sh fcoe database
Nexus 5k doesn't support AL (Arbitrated Loop) topology
sh system internal dcbx info interface e1/17
> Look for DCX Protocol where CEE means Gen2 or Gen2+ CNA and anything else like CIN means Gen1 CNA
FCOE traffic is marked with COS-3 by default
SAN gets 50% of BW incase of congestion and rest 50% is reserved for LAN
sh int fcoe
Further Readings:
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/513_n1_1/n5k_ops_fcoe.html
HTH...
Deepak Arora
Evil CCIE
Node Port Virtualization & Node Port ID Virtualization (NPV & NPIV)
+++++++++++++++++++++++++++++++++++++++++++++++++
- FCID is a 3 byte field with Domain ID as first byte
- Fibre Channel forwarding is based on FCID
- Domain ID is used to identify the Switch in the Fabric's SPT
- It implies that hard limit of switches per fabric is 256
- Some IDs are reserved so only 239 are usable but Qualified limit by OSMs
(Original Storage Manufacturer) is approx 50
- NPV fixes the Domain ID problem by removing the need for a switch to participate in Fabric Services
> I.e. no FSPF, FCNS, Zoning etc
- Switches running NPV appears to the rest of the fabric as an end host I.e. a Node Port (N_Port)
- Upstream facing link on the NPV switch is called NP_Port AKA Proxy Node Port
FC Switch/NPV Core Swith (F_Port)----------(NP_Port) NPV Switch (F_Port)----------(N_Port) Initiator
- Switch upstream of NPV switch is the NPV core switch
- NPV Core switch runs NPIV
- NPIV allows multiple FLOGIs and FCID assignments on its F Port facing downstream
======================================================
NPV/NPIV Configuration
+++++++++++++++++++++++
- Enable NPV on NPV Switch (Downstream Switch)
> feature fcoe
> feature npv
# After enabling feature NPV,Switch would require reload. Also most of older config
including Data Plane will be erased
# On 5500 UP, reallocate ports as FC after first reload which would again require second reload
- Now configure NP Ports on NPV Switch
> switchport mode np
- Configure F Ports on NPV Switch (Facing Initiator) / NPIV Switch (Facing NPV Switch)
> switchport mode f
- ENable NPIV on the Core Swith
> feature npiv
# sh npv flogi-table ( To check Initiators flogi on NPV Switch )
# sh npv external-interface-usage (To check F port to NP Port Mapping on NPV SW to
verify static pinning distribution)
- Zoning to be configured on NPIV Switch
HTH...
Deepak Arora
Evil CCIE
Fiber Channel Over IP (FCIP)
++++++++++++++++++++++++++++
- SCSI over FCP over TCP over IP
- Same protocol stack as fiber channel
- Initiator and Targets are still Native FC (or FCOE)
- Used for FC SAN Extension like SAN replication over DCI (Data Center Interconnect),
for example running FCIP over OTV will be - SCSI over FCP over TCP over IP over
Ethernet over MPLS over GRE over IP over Ethernet
===============================================================
MDS FCIP Gateway Configuration
+++++++++++++++++++++++++++++++
- Configure normal FC to initiators & targets
- Configure IP Connectivity between MDSes
- Configure FCIP Tunnel
- FCIP tunnel now counts as a TE port
_ Normal FC Switching design now applies
- FCIP only supported on MDS
================================================================
FCIP Configuration Example
++++++++++++++++++++++++++
!
feature fcip
!
fcip profile 10
ip add 1.1.1.1
exit
!
inteface fcip 12
use-profile 10
peer-info ipadd 1.1.1.2
no shut
exit
!
sh fcip summary
sh int fcip 12 brief
sh int fcip12 trunk vsan
sh fcip profile
HTH...
Deepak Arora
Evil CCIE
+ iSCSI ( Internet Small Computer System Interface )
####################################################
- Completely separate protocol stack from Fibre channel
- Typically used in small to mid range SANs
- No dedicated SAN switches required which implies no SAN switching knowledge required
- 1/10 GigE iSCSI hardware offload cards available
- End host/Storage Array just runs IP
- Transport supports IP and can be of any IP Transport type like Ethernet, Frame Relay or Token Ring
- MDS is an iSCSI to FC Gateway, so MDS is a translational bridge for Fiber Channel & iSCSI
=====================================
+ iSCSI Gateway Operation
########################
- FC Targets FLOGI to FC Fabric
- iSCSI initiators send Discovery to MDS using Ethernet for example
- MDS applies zoning/Access Lists
- iSCSI initiator things FC target is iSCSI target
- FC Target thinks iSCSI initiator is a FC initiator
=====================================
+ MDS iSCSI Gateway Configuration Steps
#######################################
- Configure FC to Targets
- Configure IP to Initiators
- Enable iSCSI
- Configure ZONING/ Access Control
- Point Server at MDS's IP Address
=====================================
+ Access Control in iSCSI
##########################
- Access Control can be enforced as Zoning based upon :
> pwwn, fcid, alias
> Initiator's IP Address
> Initiator's iSCSI qualified name (IQN),IQNs are generated automatically by initiator
but can be configured manually.
> iSCI based virtual target to present LUN based on IQN or IP Address/Subnet
=====================================
sh int fc1/20 trunk vsan
feature iscsi
iscsi enable module 1
int iscsi 1/1
no sh
inc iscsi 1/2
no sh
iscsi import target fc
sh iscsi global
sh iscsi initiator
HTH...
Deepak Arora
Evil CCIE
FC Aliases
++++++++++
- FC Aliases gives user friendly names to the WWNs, FCIDs etc and are analogous to DNS in IP
- Syntex - fcalias name
- Alias can be advertised through Zoneset distribution. Syntex - zoneset distribute vsan 1
- Example:
!
fcalias name ABC vsan 30
member pwwn
exit
!
- Verification: sh fcalias
- Creating zone and Zoneset using ALIAS:
!
zone name ABC vsan 30
member fcalias DISK1
member fcalias DISK2
member fcalias DISK3
member fcalias SERVER
exit
!
!
zoneset name XYZ vsan 30
member ABC
!
!
zoneset activate name XYZ vsan 30
!
+++++++++++++++++++++++++++++++++++
Basic Vs Enhanced Zoning
########################
- By default the full zoneset is local and the active zoneset is fabric-wide
- Order of operations errors can corrupt the Active Zoneset
- Enchanced zoning prevents this by Locking the fabric which ensures people don't
accidentally overwrite each other
- Enhanced Zoning works on per VSAN basis
- The lock on zone is released by "committing" the zoneset
- But if Admin forgets to commit, someone else can release lock as well with
"clear zone lock vsan "
- Enabling Enhanced Zoning
> zone mode enhanced vsan < Per VSAN Basis
> system default zone mode enhanced < For Entire System
- Configuring Enhanced Zoning doesn't change anything for regular FCALIAS and
they still remain local by default
- To solve this problem solution is "Device Aliases"
- Device Aliases serves the same purpose as FC Aliases by binding PWWNs to a user
friendly name but the difference
is that the binding is advertised to the fabric and doesn't remain local only
- Configured as device-alias database
- Changing the zoning mode from normal to enhanced is not disruptive
- Device aliases are advertised through CFS so should not be used in multi vendor environment
+ Configured as "device-alias database"
+ device-alias commit
E.g.>
device-alias database
device-alias name ABC_Server pwwn
device-alias name XYZ_Disk1 pwwn
device-alias name XYZ_Disk2 pwwn
device-alias name XYZ_Disk3 pwwn
device-alias commit
show device-alias database
zone name ABC vsan
member device-alias ABC_Server
member device-alias XYZ_Disk1
member device-alias XYZ_Disk2
zoneset name XYZ vsan
member ABC
exit
zoneset activate name XYZ vsan
sh zoneset pending
zone commit vsan
sh zone active
"sh port-resource module " shows port grouping on MDS
(shared vs dedicaed mode), in 5k all ports are in dedicated mode by default.
HTH...
Deepak Arora
Evil CCIE
Zoning
######
- Controls which initiators can talk to which targets
- Zoing in required, not optional
- Default zone policy is to deny everything in Data Plane
- Default zone policy can be changed per VSAN or for entire system but not recommended
> zone default-zone permit vsan 1
> system default zone
# Soft Zoning
=============
- Initiator registers with FCNS to get zoning
- Zoning enforced in the control plane but not data plane
- Initiator could manually mount the wrong target if Target FCID is known in prior
# Hard Zoning
=============
- Initiator registers with FCNS to get zoning
- Zoning enforced in the control plane & data plane
- Initiator cannot manually mount the wrong target if Target FCID is known in prior
Hard Zoning is default in NX-OS and SAN-OS
# Zone Vs Zoneset
=================
- Zone is used to create a mapping for access based upon WWPNs, FCIDs, Aliases,
Interface, Domain-ID etc
- Zones are then grouped together in a Zoneset
> Zoneset is overall ACL and Zone is an Access Control Entry
- Zones are applied to VSAN and then activated
# If zone is just configured and not activated - it's called a Zone Set. But when activated
it's called Active Zone Set. So this is how Zone set Vs Active zone set could be different
# Any changes made into Zoning doesn't go into affect unless Zoneset is re-activated
- Only one zoneset per VSAN can be "Active" in the fabric at a time
- By default only Active Zone set is advertised into the fabric not the full zone set which
can cause Isolation between Switches because of having different copies of zone sets.
- sh zone status vsan 1
- sh zone
- sh zone active
- sh zoneset
- sh zoneset active
- clear zone database vsan 1
- zoneset distribute full vsan 1 ( Global mode)
- zoneset distribute vsan 1 (Exec Mode)
# Zone Config
==============
zone name ABC vsan 30 < Create Zone
attribute read-only < Only Supported on MDS
member pwnn
member pwnn
exit
zoneset name XYZ vsan 30 < Create Zoneset
member ABC
exit
zoneset activate name XYZ vsan 30 < Activate ZoneSet
HTH...
Deepak Arora
Evil CCIE
Sample SAN Port Channel Config
==============================
interface san-port-channel 1 >> In Nexus # in MDS >> interface port-channel
channel mode active
switchport mode E
swithcport trunk allowed vsan 1
switchport trunk allowed vsan add 10
switchport speed 4000
!
int x/x
channel-group 1 > Static port channel
channel-group 1 foce > To add any link later into functional San Port Channel
channel-mode active > To enable port channel protocol
Verification
============
sh port-channel summary
sh san-port-channel summary
sh port-channel internal info interface po1
sh interface san-port-channel 1 trunk vsan
=============================================================================
This is order of operations for E/TE:
1. Go to the MDS physical interfaces, shut them down
2. Need to ensure the individual ports are dedicated - 'switchport rate mode dedicated'
otherwise will error out on port-channel configuration. Can also configure
'switchport mode e'
3. Add ports to port-channel - 'channel-group X'
4. Go to the port-channel, apply the configuration ('switchport mode e' and
'switchport rate mode dedicated'.
5. If required, restrict the port-channel with the only the necessary VSANs
6. Add the port-channel to the VSAN database (optional - again depends on requirements)
7. No shut the port-channel and then physical ports
* Make use of 'show interface fc x/y' and verify config - look at: Admin port mode,
trunk mode, port mode, port vsan, VSANs allowed (if trunking).
* Ensure that these settings match on both sides (either MDS to N5K or UCS)
For F/TF port-channel, steps that are slightly different from above:
1. Enable MDS features (such as npiv / fport-channel-trunk - if making a TF port-channel)
2. For F ports, can leave in either shared or dedicated mode. Additionally, configure the
physical interfaces as 'switchport mode f', otherwise will error out as generally
they default to FX ports
3. Instead of 'switchport mode e' on the port-channel, obviously make it an F port
If link fails to come up and using NPV-NPiV, useful to make use of 'show npv status',
will also identify if you've forgotten to enable NPiV upstream, etc.
HTH...
Deepak Arora
Evil CCIE
Principle Switch (PS) Election
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Election starts when E port between two or more switches comes up
# Assign Domain IDs automatically but can be assigned manually as well
with - ( fcdomain domain 0x51 vsan )
- Preferred > Switch asks principal switch if it can use static domain id, if PS
says "Yes" than fine, If "NO" it uses Domain ID given by PS
- Static > Same theory as above but if PS says NO, the switch gets isolated from the FC network
Election based on
++++++++++++
- Lowest priority ( fcdomain priority 1 vsan 1)
- Lowest WWN ( sh wwn switch)
* sh fcdomain (vsan)
* sh fcdomain domain-list (vsan)
FC Domain Services Restart
+++++++++++++++++++
- Graceful - fcdomain restart vsan 1
- Forced - fcdomain restart disruptive vsan 1 < Hidden command
Everytime we make changes into FC Services, the services must be restarted
==============================================================
# Note: One copy of FC services runs on per VSAN basis which means One
domain id can be used for multiple VSANs but it doesn't affect the design.
It's more like using one OSPF router id for multiple ospf processes. But each
process populates a separate Datbase which is completely unrelated from each other.
==============================================================
Fabric Shortest Path First (FSPF)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Used to build an SPT through the fabric
- Domain ID is the Node ID in the SPT calculation
- FSPF runs automatically on per VSAN basis
- FSPF perameters can be manually modified in global config mode with
"fspf config vsan 1" or on interface level like
" fspf [cost|dead-interval|hello-interval|passive|retransmit-interval]
Verification
++++++++++++
- sh fspf [database|interface] [vsan]
- sh fcroute unicast vsan
==============================================================
Fabric Login (FLOGI)
^^^^^^^^^^^^^^^^^^^^
- All initiators and targets must FLOGI before sending any data into the fabric
- Verification - sh flogi database vsan
- No configuration required
- No FLOGI indicates a basic link-level negotiation problem
- The FLOGI database is local to switch which means only directly connected devices
will show up, It's FCNS job to progpogate this information to entire fabric
==============================================================
Fiber Channel Name Services
^^^^^^^^^^^^^^^^^^^^^^^^^^^
- FCNS (AKA Directory Services) keeps a mapping of FCIDs to WWPNs
- Analogous to IP ARP Cache in ethernet
- End devices register with the FCNS after FLOGI
- sh fcns database
- If Node did FLOGI but is not in everyone's FCNS, it indicates the Fabric is broken
> E.g. VSAN is isolated, EISL allowed list is wrong etc
After couple of months break now I am back on my CCIE DC preparation track. Since I was pretty much done with Nexus at Initial level and also did couple of Nexus Installations at work along with OTV deployment I thought to lets start with storage now. Since I had no experience with storage at any level earlier so I thought to start with baby steps as of now and catch up the pace later.
When it comes to storage there is no better choice other to start learning one of Storage Core Protocol known as Fibre Channel AKA FC.
So I did some initial FC study yesterday and compiled some notes for reference. Hope you guys will find it helpful to begin with storage.
######################################################################
Fibre Channel (FC)
==================
> FC is a Protocol Stack primarily used to send SCSI commands over the SAN. But we can also run IP/ARP over FC
# FC Suppported topologies
1. P2P (FC-P2P)
2. Arbitrated Loop (FC-AL) similar to Token Ring
3. Switched Fabric (FC-SW) similar to Ethernet
# FC Port Types
1. Node Port AKA N Port - End host (Target or initiator) in P2P or Switched Fabric
2. Node Loop Port or NL Port - Similar to N port but in Arbitrated Loop Topology e.g. > JBOD Connectivity
3. Fabric Port or F Port - Switch port connecting to Node Port
4. Fabric Loop or FL Port - Switch port connecting to Node Loop Port
5. Expansion Port or E Port aka ISL = Link between two switches
6. Trunk Expension Port or TE Port = Link between two switches to carry VSANs similar to DOT1Q trunk in LAN
# Fibre Channel Addressing
1. Physical Address = World Wide Names (WWNs) is equal to MAC in Ethernet = 8 Bytes
2. Logical Address = FC Identifier (FC ID) is equal to IP address = 3 Bytes
* WWN is subdivided into two parts
1. World Wide Node Name (WWNN)
- Switch, Server or Disk's Physical Address
2. World Wide Port Name (WWPN)
- Switch, Server or Disk's Port's Physical Address
E.g. > A HBA with multiple interfaces. HBA will have one WWNN and Each HBA port will have it's own WWPN
Note - WWN is not used in Data Plane
* FCID is subdivided into three fields
1. Domain ID
- Each switch gets a separate domain id
2. Area ID
- Group of ports on switch have an area id
3. Port ID
- End station connected to switch gets a Port ID
> Domain ID is automatically assigned by Principal Switch but can also be manually assigned. Principal switch is analogous to STP root bridge in Ethernet world.
# Fabric Shortest Path First (FSPF) is protocol in SAN used to route traffic between
Switches based on FCID's Domain ID
FSPF support ECMP and used Dijkstra Algorithm like OSPF & ISIS and by default enabled on SAN Switch as Service
# FC supports SAN Port Channel using Port Channel Protocol (PCP) roghly analogous to 802.3ad
# In Fibre Channel network all end hosts must first register with the control plane of fabric before sending any traffic
# Fabric Registration has three parts
1. Fabric Login (FLOGI)
2. Port Login (PLOGI)
3. Process Login (PLRI)
FLOGI
++++++
> During FLOGI, the N Port tells switch's F port that it wants to register
> Once Switch gets Flogi Request, it learns the WWNN and WWPN of node
> Later switch assigns FCID to node
PLOGI
+++++
> End to End Login between N Ports
> Initiator (End Device such as server) tells target (Storage Array) that it wants to talk
> Used for applications such as End To End flow control
PLRI
+++++
> Upper Layer protocol login negotiation between N ports
# Fiber Channel Name Server (FCNS) is analogous to ARP cache and helps in FLOGI.
> FCNS is used to resolve WWN to FCID
sh fcns database
sh flogi database
ZONING
======
> Zoning controls which initiators talks to which targets and alalogous to ACL in IP World
- Associates WWNs, FCIDs, aliases etc to control who can talk to whom
Virtual SANs (VSANs)
====================
> VSANs are roughly analogous to VLANs in Ethernet world
> VSANs are used to create small SAN separations within large SAN. Also can be used to isolate the management and failure domain of the network
> Creating a VSAN creates a separate/new set of Fabric Services like FLOGI, FCNS, ZONING, Aliases etc
> Normally we don't create more than two VSANs
> With VSANs, E Port now becomes TE port
HTH...
Deepak Arora
Evil CCIE
