FC Aliases
++++++++++
- FC Aliases gives user friendly names to the WWNs, FCIDs etc and are analogous to DNS in IP
- Syntex - fcalias name
- Alias can be advertised through Zoneset distribution. Syntex - zoneset distribute vsan 1
- Example:
!
fcalias name ABC vsan 30
member pwwn
exit
!
- Verification: sh fcalias
- Creating zone and Zoneset using ALIAS:
!
zone name ABC vsan 30
member fcalias DISK1
member fcalias DISK2
member fcalias DISK3
member fcalias SERVER
exit
!
!
zoneset name XYZ vsan 30
member ABC
!
!
zoneset activate name XYZ vsan 30
!
+++++++++++++++++++++++++++++++++++
Basic Vs Enhanced Zoning
########################
- By default the full zoneset is local and the active zoneset is fabric-wide
- Order of operations errors can corrupt the Active Zoneset
- Enchanced zoning prevents this by Locking the fabric which ensures people don't
accidentally overwrite each other
- Enhanced Zoning works on per VSAN basis
- The lock on zone is released by "committing" the zoneset
- But if Admin forgets to commit, someone else can release lock as well with
"clear zone lock vsan "
- Enabling Enhanced Zoning
> zone mode enhanced vsan < Per VSAN Basis
> system default zone mode enhanced < For Entire System
- Configuring Enhanced Zoning doesn't change anything for regular FCALIAS and
they still remain local by default
- To solve this problem solution is "Device Aliases"
- Device Aliases serves the same purpose as FC Aliases by binding PWWNs to a user
friendly name but the difference
is that the binding is advertised to the fabric and doesn't remain local only
- Configured as device-alias database
- Changing the zoning mode from normal to enhanced is not disruptive
- Device aliases are advertised through CFS so should not be used in multi vendor environment
+ Configured as "device-alias database"
+ device-alias commit
E.g.>
device-alias database
device-alias name ABC_Server pwwn
device-alias name XYZ_Disk1 pwwn
device-alias name XYZ_Disk2 pwwn
device-alias name XYZ_Disk3 pwwn
device-alias commit
show device-alias database
zone name ABC vsan
member device-alias ABC_Server
member device-alias XYZ_Disk1
member device-alias XYZ_Disk2
zoneset name XYZ vsan
member ABC
exit
zoneset activate name XYZ vsan
sh zoneset pending
zone commit vsan
sh zone active
"sh port-resource module " shows port grouping on MDS
(shared vs dedicaed mode), in 5k all ports are in dedicated mode by default.
HTH...
Deepak Arora
Evil CCIE
Zoning
######
- Controls which initiators can talk to which targets
- Zoing in required, not optional
- Default zone policy is to deny everything in Data Plane
- Default zone policy can be changed per VSAN or for entire system but not recommended
> zone default-zone permit vsan 1
> system default zone
# Soft Zoning
=============
- Initiator registers with FCNS to get zoning
- Zoning enforced in the control plane but not data plane
- Initiator could manually mount the wrong target if Target FCID is known in prior
# Hard Zoning
=============
- Initiator registers with FCNS to get zoning
- Zoning enforced in the control plane & data plane
- Initiator cannot manually mount the wrong target if Target FCID is known in prior
Hard Zoning is default in NX-OS and SAN-OS
# Zone Vs Zoneset
=================
- Zone is used to create a mapping for access based upon WWPNs, FCIDs, Aliases,
Interface, Domain-ID etc
- Zones are then grouped together in a Zoneset
> Zoneset is overall ACL and Zone is an Access Control Entry
- Zones are applied to VSAN and then activated
# If zone is just configured and not activated - it's called a Zone Set. But when activated
it's called Active Zone Set. So this is how Zone set Vs Active zone set could be different
# Any changes made into Zoning doesn't go into affect unless Zoneset is re-activated
- Only one zoneset per VSAN can be "Active" in the fabric at a time
- By default only Active Zone set is advertised into the fabric not the full zone set which
can cause Isolation between Switches because of having different copies of zone sets.
- sh zone status vsan 1
- sh zone
- sh zone active
- sh zoneset
- sh zoneset active
- clear zone database vsan 1
- zoneset distribute full vsan 1 ( Global mode)
- zoneset distribute vsan 1 (Exec Mode)
# Zone Config
==============
zone name ABC vsan 30 < Create Zone
attribute read-only < Only Supported on MDS
member pwnn
member pwnn
exit
zoneset name XYZ vsan 30 < Create Zoneset
member ABC
exit
zoneset activate name XYZ vsan 30 < Activate ZoneSet
HTH...
Deepak Arora
Evil CCIE
Sample SAN Port Channel Config
==============================
interface san-port-channel 1 >> In Nexus # in MDS >> interface port-channel
channel mode active
switchport mode E
swithcport trunk allowed vsan 1
switchport trunk allowed vsan add 10
switchport speed 4000
!
int x/x
channel-group 1 > Static port channel
channel-group 1 foce > To add any link later into functional San Port Channel
channel-mode active > To enable port channel protocol
Verification
============
sh port-channel summary
sh san-port-channel summary
sh port-channel internal info interface po1
sh interface san-port-channel 1 trunk vsan
=============================================================================
This is order of operations for E/TE:
1. Go to the MDS physical interfaces, shut them down
2. Need to ensure the individual ports are dedicated - 'switchport rate mode dedicated'
otherwise will error out on port-channel configuration. Can also configure
'switchport mode e'
3. Add ports to port-channel - 'channel-group X'
4. Go to the port-channel, apply the configuration ('switchport mode e' and
'switchport rate mode dedicated'.
5. If required, restrict the port-channel with the only the necessary VSANs
6. Add the port-channel to the VSAN database (optional - again depends on requirements)
7. No shut the port-channel and then physical ports
* Make use of 'show interface fc x/y' and verify config - look at: Admin port mode,
trunk mode, port mode, port vsan, VSANs allowed (if trunking).
* Ensure that these settings match on both sides (either MDS to N5K or UCS)
For F/TF port-channel, steps that are slightly different from above:
1. Enable MDS features (such as npiv / fport-channel-trunk - if making a TF port-channel)
2. For F ports, can leave in either shared or dedicated mode. Additionally, configure the
physical interfaces as 'switchport mode f', otherwise will error out as generally
they default to FX ports
3. Instead of 'switchport mode e' on the port-channel, obviously make it an F port
If link fails to come up and using NPV-NPiV, useful to make use of 'show npv status',
will also identify if you've forgotten to enable NPiV upstream, etc.
HTH...
Deepak Arora
Evil CCIE
Principle Switch (PS) Election
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Election starts when E port between two or more switches comes up
# Assign Domain IDs automatically but can be assigned manually as well
with - ( fcdomain domain 0x51 vsan )
- Preferred > Switch asks principal switch if it can use static domain id, if PS
says "Yes" than fine, If "NO" it uses Domain ID given by PS
- Static > Same theory as above but if PS says NO, the switch gets isolated from the FC network
Election based on
++++++++++++
- Lowest priority ( fcdomain priority 1 vsan 1)
- Lowest WWN ( sh wwn switch)
* sh fcdomain (vsan)
* sh fcdomain domain-list (vsan)
FC Domain Services Restart
+++++++++++++++++++
- Graceful - fcdomain restart vsan 1
- Forced - fcdomain restart disruptive vsan 1 < Hidden command
Everytime we make changes into FC Services, the services must be restarted
==============================================================
# Note: One copy of FC services runs on per VSAN basis which means One
domain id can be used for multiple VSANs but it doesn't affect the design.
It's more like using one OSPF router id for multiple ospf processes. But each
process populates a separate Datbase which is completely unrelated from each other.
==============================================================
Fabric Shortest Path First (FSPF)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- Used to build an SPT through the fabric
- Domain ID is the Node ID in the SPT calculation
- FSPF runs automatically on per VSAN basis
- FSPF perameters can be manually modified in global config mode with
"fspf config vsan 1" or on interface level like
" fspf [cost|dead-interval|hello-interval|passive|retransmit-interval]
Verification
++++++++++++
- sh fspf [database|interface] [vsan]
- sh fcroute unicast vsan
==============================================================
Fabric Login (FLOGI)
^^^^^^^^^^^^^^^^^^^^
- All initiators and targets must FLOGI before sending any data into the fabric
- Verification - sh flogi database vsan
- No configuration required
- No FLOGI indicates a basic link-level negotiation problem
- The FLOGI database is local to switch which means only directly connected devices
will show up, It's FCNS job to progpogate this information to entire fabric
==============================================================
Fiber Channel Name Services
^^^^^^^^^^^^^^^^^^^^^^^^^^^
- FCNS (AKA Directory Services) keeps a mapping of FCIDs to WWPNs
- Analogous to IP ARP Cache in ethernet
- End devices register with the FCNS after FLOGI
- sh fcns database
- If Node did FLOGI but is not in everyone's FCNS, it indicates the Fabric is broken
> E.g. VSAN is isolated, EISL allowed list is wrong etc
As I continue my journey with storage networking, I compiled some more notes today covering Nexus 5K initialization from Fiber Channel Implementation Perspective. Hope you guys will enjoy !!!
######################################################################
* To verify SAN Port type and topology on N5k => sh int brief
+ In output look under " Oper Mode "
Sample SAN Port Channel Config
==============================
interface san-port-channel 1
channel mode active
switchport mode E
swithcport trunk allowed vsan 1
switchport trunk allowed vsan add 10
switchport speed 4000
* Changing Port Type to FC in 5k
================================
slot 1 or 2
port 25 - 32 type fc
+ Requires Reload
# " feature fcoe " needs to be turned on separately in order to enable basic FC protocol
which also further enables actual FCOE. Though NPV for FCOE requires separate license.
+ If port type has been changed to FC but feature fcoe is not enabled, the FC ports
will not show up in "sh int brief" output. Also we will not be able to go into FC interface
configuration mode.
+ The SFP for storage connecivity is different from SFP for Ethernet connection from 5K standpoint
* FC Auto Negotiation
=====================
> FC Auto Negotiation is enabled by default
> Auto Negotiation Determines:
+ Port Type [ switchport mode ]
+ Trunk Mode [ switchport trunk mode ]
+ Speed [ switchport speed <1000> ]
- From real world standpoint, Port Types etc should be configured manually
* MDS Auto-Negotiation
======================
- MDS port types are by default FX which means it can figure out if it needs to be F or FL
port mode during auto-negotiation
- MDS usually can't negotiate E ports so ISL must always be configured manually.
- In MDS E port requires dedicated rate-mode [switchport rate-mode dedicated ]
+ In FC network it's not only SFP type that will control the link speed but also Optical
cable should be of same speed in order to match it
+ In FC network there is something known as B2B or Buffer To Buffer credit which is
analogous to Flow Control is Ethernet Networks.B2B works per link basis and on end
to end basis between initiator & target
* FC Trunk Protocol
===================
- Trunk protocol (TP) in FC is analogous to DTP in ethernet and is enabled by default
- Verification - sh trunk protocol
- To disable TP - switchport trunk mode on (Default - auto)
But keep in mind there is no command called "switchport mode te" to statically configure TE port
