Fibre Channel Over Ethernet
+++++++++++++++++++++++++++
- Unified Fabric = Unified Wire = Converged Ethernet = Data Center Ethernet = Data Center Bridging
- In unified fabric - Ethernet & Fibre channel both run on common infrastructure
- FCoE Initialization Protocol (FIP) is control plane protocol for FCoE running between Initiator & FCOE Switch/FCoE Forwarder(FCF)
- ENode in FCoE = N port in FC
- Virtual Fibre Channel (VFC) interface is logical interface mapped with physical interfaces. FIP Runs between ENode & VFC using P2P Adjacency
- Virtual Port Types in FCOE
> VN Port = N Port in FC
> VF Port = F Port in FC
> VE Port = E Port in FC < Used in Multi Hop FCOE
> VTE Port = TE Port in FC
- FCoE replaces layer 1 & layer 2 tranport for FC but all upper layer FC services remains same like Zoning, Domain IDs, FSPF, FLOGI, FCNS etc
- FIP is control place of FCoE and FCoE itself is the actual data plane
# FIP
++++++
- FIP has new Ether Type as 0x8914
- FIP is used to discover FCFs and perform FLOGI
# FCOE
+++++++
- New Ethertype 0x8906
- Max length of 2240 bytes which implies jumbo frames are required
www.t11.org/fcoe
# FCoE Addressing
+++++++++++++++++
- Fibre Channel uses 3 bytes FCIDs
- Ethernet uses 6 byte MAC Addresses
- FCoE ENode gets a Fabric Provided MAC Address (FPMA) for FCoE
- During FIP, ENode is assigned a 3 byte FCID
- FLOGI now is part of FIP
- FCF is configured with a 3 byte FCoE MAC Address Prefix (FC-MAP)
- ENode appends FC-MAP to FCID
- 3 byte FC-MAP + 3 byte FCID = FPMA
# Configuring FCOE
+++++++++++++++++++
- feature fcoe < Feature FCOE can only be enabled from Admin VDC in Nexus 7k
- feature lldp
- Create a VSAN
vsan database
vsan 1010
vsan 1010 interface fc1/10
- Associate VSAN to a VLAN
vlan 1010
fcoe vlan 1010
- Configure a VFC
- Associate Physical Ethernet to a VFC
int vfc117
bind interface e1/17
switchport trunk allowed vsan 1010
- Assign VFC to the VSAN
vsan database
vsan 1010 interface vfc117
- Configure physical ethernet as a trunk
int e1/17
shut
switchport mode trunk
spanning-tree port type edge trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 10 ,1010
- Activate Interfaces
int vfc117
no shut
int e1/17
no shut
sh int vfc117 < Look for trunking & VSAN should be up
sh fcoe database
Nexus 5k doesn't support AL (Arbitrated Loop) topology
sh system internal dcbx info interface e1/17
> Look for DCX Protocol where CEE means Gen2 or Gen2+ CNA and anything else like CIN means Gen1 CNA
FCOE traffic is marked with COS-3 by default
SAN gets 50% of BW incase of congestion and rest 50% is reserved for LAN
sh int fcoe
Further Readings:
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/513_n1_1/n5k_ops_fcoe.html
HTH...
Deepak Arora
Evil CCIE
R8#sh ip int b | e una|do
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 89.0.0.8 YES NVRAM up up
Loopback0 8.8.8.8 YES NVRAM up up
R8#sh run | s r o
router ospf 1
log-adjacency-changes
network 0.0.0.0 255.255.255.255 area 0
R9#sh ip int b | e una|do
Interface IP-Address OK? Method Status Protocol
Serial1/0 89.0.0.9 YES NVRAM up up
Loopback0 9.9.9.9 YES NVRAM up up
R9#sh run | s r o
router ospf 1
log-adjacency-changes
network 0.0.0.0 255.255.255.255 area 0
R9#sh run int s1/0
Building configuration...
Current configuration : 132 bytes
!
interface Serial1/0
ip address 89.0.0.9 255.255.255.0
encapsulation ppp
ip ospf network broadcast
serial restart-delay 0
end
R2#sh ip int b | e una|do
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 12.0.0.2 YES NVRAM up up
FastEthernet0/1 23.0.0.2 YES NVRAM up up
Loopback0 2.2.2.2 YES NVRAM up up
R2#sh run | s r e
router eigrp 100
network 0.0.0.0
no auto-summary
R2#sh ip ro e
1.0.0.0/32 is subnetted, 1 subnets
D 1.1.1.1 [90/409600] via 12.0.0.1, 00:10:30, FastEthernet0/0
3.0.0.0/32 is subnetted, 1 subnets
D 3.3.3.3 [90/409600] via 23.0.0.3, 00:10:10, FastEthernet0/1
R2#sh mpls forwarding
Tag switching is not operational.
CEF or tag switching has not been enabled.
No TFIB currently allocated.
R1#sh run | s pseudo
pseudowire-class L2TPv3
encapsulation l2tpv3
interworking ip
ip local interface Loopback0
R1#sh run int f2/1
Building configuration...
Current configuration : 124 bytes
!
interface FastEthernet2/1
no ip address
speed auto
duplex auto
no keepalive
xconnect 3.3.3.3 13 pw-class L2TPv3
end
R3#sh run | s pseudo
pseudowire-class L2TPv3
encapsulation l2tpv3
interworking ip
ip local interface Loopback0
R3#sh run int s1/1
Building configuration...
Current configuration : 122 bytes
!
interface Serial1/1
no ip address
encapsulation ppp
serial restart-delay 0
xconnect 1.1.1.1 13 pw-class L2TPv3
end
R8#sh ip ospf nei
Neighbor ID Pri State Dead Time Address Interface
9.9.9.9 1 FULL/DR 00:00:37 89.0.0.9 FastEthernet0/0
R8#sh ip ro os
9.0.0.0/32 is subnetted, 1 subnets
O 9.9.9.9 [110/11] via 89.0.0.9, 00:13:00, FastEthernet0/0
R8#ping 9.9.9.9 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.9, timeout is 2 seconds:
Packet sent with a source address of 8.8.8.8
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 88/120/180 ms
R1#sh l2tp tunnel all
L2TP Tunnel Information Total tunnels 1 sessions 1
Tunnel id 1100418086 is up, remote id is 2518749472, 1 active sessions
Remotely initiated tunnel
Tunnel state is established, time since change 00:14:48
Tunnel transport is IP (115)
Remote tunnel name is R3
Internet Address 3.3.3.3, port 0
Local tunnel name is R1
Internet Address 1.1.1.1, port 0
L2TP class for tunnel is l2tp_default_class
Counters, taking last clear into account:
111 packets sent, 107 received
9008 bytes sent, 8752 received
Last clearing of counters never
Counters, ignoring last clear:
111 packets sent, 107 received
9008 bytes sent, 8752 received
Control Ns 9, Nr 13
Local RWS 1024 (default), Remote RWS 1024
Control channel Congestion Control is disabled
Tunnel PMTU checking disabled
Retransmission time 1, max 1 seconds
Unsent queuesize 0, max 0
Resend queuesize 0, max 1
Total resends 0, ZLB ACKs sent 11
Total out-of-order dropped pkts 0
Total out-of-order reorder pkts 0
Total peer authentication failures 0
Current no session pak queue check 0 of 5
Retransmit time distribution: 0 0 0 0 0 0 0 0 0
Control message authentication is disabled
R1#sh l2tun session all
L2TP Session Information Total tunnels 1 sessions 1
Session id 3278375310 is up, logical session id 65537, tunnel id 1100418086
Remote session id is 1558878854, remote tunnel id 2518749472
Remotely initiated session
Unique ID is 1
Session Layer 2 circuit, type is Ethernet, name is FastEthernet2/1
Session vcid is 13
Interworking type is IP
Circuit state is UP
Local circuit state is UP
Remote circuit state is UP
Call serial number is 1168000001
Remote tunnel name is R3
Internet address is 3.3.3.3
Local tunnel name is R1
Internet address is 1.1.1.1
IP protocol 115
Session is L2TP signaled
Session state is established, time since change 00:15:43
117 Packets sent, 113 received
9488 Bytes sent, 9232 received
Last clearing of counters never
Counters, ignoring last clear:
117 Packets sent, 113 received
9488 Bytes sent, 9232 received
Receive packets dropped:
out-of-order: 0
other: 0
total: 0
Send packets dropped:
exceeded session MTU: 0
other: 0
total: 0
DF bit off, ToS reflect disabled, ToS value 0, TTL value 255
Sending UDP checksums are disabled
Received UDP checksums are verified
No session cookie information available
FS cached header information:
encap size = 24 bytes
45000014 00000000 ff73b36f 01010101
03030303 5cea9a86
Sequencing is off
Conditional debugging is disabled
SSM switch id is 4096, SSM segment id is 8193
Final Config - http://www.4shared.com/rar/T4TwmRG7/Interworking.html
Further Readings:
http://www.cisco.com/en/US/docs/ios-xml/ios/mp_l2_vpns/configuration/15-2mt/mp-l2vpn-intrntwkg.html#GUID-E774CB40-066C-4B3F-8E1E-BEBCBED1087C
http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/fsinterw.html#wp1057606
http://blog.ine.com/2008/01/28/poor-mans-vpls/
HTH...
Deepak Arora
Evil CCIE
Node Port Virtualization & Node Port ID Virtualization (NPV & NPIV)
+++++++++++++++++++++++++++++++++++++++++++++++++
- FCID is a 3 byte field with Domain ID as first byte
- Fibre Channel forwarding is based on FCID
- Domain ID is used to identify the Switch in the Fabric's SPT
- It implies that hard limit of switches per fabric is 256
- Some IDs are reserved so only 239 are usable but Qualified limit by OSMs
(Original Storage Manufacturer) is approx 50
- NPV fixes the Domain ID problem by removing the need for a switch to participate in Fabric Services
> I.e. no FSPF, FCNS, Zoning etc
- Switches running NPV appears to the rest of the fabric as an end host I.e. a Node Port (N_Port)
- Upstream facing link on the NPV switch is called NP_Port AKA Proxy Node Port
FC Switch/NPV Core Swith (F_Port)----------(NP_Port) NPV Switch (F_Port)----------(N_Port) Initiator
- Switch upstream of NPV switch is the NPV core switch
- NPV Core switch runs NPIV
- NPIV allows multiple FLOGIs and FCID assignments on its F Port facing downstream
======================================================
NPV/NPIV Configuration
+++++++++++++++++++++++
- Enable NPV on NPV Switch (Downstream Switch)
> feature fcoe
> feature npv
# After enabling feature NPV,Switch would require reload. Also most of older config
including Data Plane will be erased
# On 5500 UP, reallocate ports as FC after first reload which would again require second reload
- Now configure NP Ports on NPV Switch
> switchport mode np
- Configure F Ports on NPV Switch (Facing Initiator) / NPIV Switch (Facing NPV Switch)
> switchport mode f
- ENable NPIV on the Core Swith
> feature npiv
# sh npv flogi-table ( To check Initiators flogi on NPV Switch )
# sh npv external-interface-usage (To check F port to NP Port Mapping on NPV SW to
verify static pinning distribution)
- Zoning to be configured on NPIV Switch
HTH...
Deepak Arora
Evil CCIE
Fiber Channel Over IP (FCIP)
++++++++++++++++++++++++++++
- SCSI over FCP over TCP over IP
- Same protocol stack as fiber channel
- Initiator and Targets are still Native FC (or FCOE)
- Used for FC SAN Extension like SAN replication over DCI (Data Center Interconnect),
for example running FCIP over OTV will be - SCSI over FCP over TCP over IP over
Ethernet over MPLS over GRE over IP over Ethernet
===============================================================
MDS FCIP Gateway Configuration
+++++++++++++++++++++++++++++++
- Configure normal FC to initiators & targets
- Configure IP Connectivity between MDSes
- Configure FCIP Tunnel
- FCIP tunnel now counts as a TE port
_ Normal FC Switching design now applies
- FCIP only supported on MDS
================================================================
FCIP Configuration Example
++++++++++++++++++++++++++
!
feature fcip
!
fcip profile 10
ip add 1.1.1.1
exit
!
inteface fcip 12
use-profile 10
peer-info ipadd 1.1.1.2
no shut
exit
!
sh fcip summary
sh int fcip 12 brief
sh int fcip12 trunk vsan
sh fcip profile
HTH...
Deepak Arora
Evil CCIE
+ iSCSI ( Internet Small Computer System Interface )
####################################################
- Completely separate protocol stack from Fibre channel
- Typically used in small to mid range SANs
- No dedicated SAN switches required which implies no SAN switching knowledge required
- 1/10 GigE iSCSI hardware offload cards available
- End host/Storage Array just runs IP
- Transport supports IP and can be of any IP Transport type like Ethernet, Frame Relay or Token Ring
- MDS is an iSCSI to FC Gateway, so MDS is a translational bridge for Fiber Channel & iSCSI
=====================================
+ iSCSI Gateway Operation
########################
- FC Targets FLOGI to FC Fabric
- iSCSI initiators send Discovery to MDS using Ethernet for example
- MDS applies zoning/Access Lists
- iSCSI initiator things FC target is iSCSI target
- FC Target thinks iSCSI initiator is a FC initiator
=====================================
+ MDS iSCSI Gateway Configuration Steps
#######################################
- Configure FC to Targets
- Configure IP to Initiators
- Enable iSCSI
- Configure ZONING/ Access Control
- Point Server at MDS's IP Address
=====================================
+ Access Control in iSCSI
##########################
- Access Control can be enforced as Zoning based upon :
> pwwn, fcid, alias
> Initiator's IP Address
> Initiator's iSCSI qualified name (IQN),IQNs are generated automatically by initiator
but can be configured manually.
> iSCI based virtual target to present LUN based on IQN or IP Address/Subnet
=====================================
sh int fc1/20 trunk vsan
feature iscsi
iscsi enable module 1
int iscsi 1/1
no sh
inc iscsi 1/2
no sh
iscsi import target fc
sh iscsi global
sh iscsi initiator
HTH...
Deepak Arora
Evil CCIE
FC Aliases
++++++++++
- FC Aliases gives user friendly names to the WWNs, FCIDs etc and are analogous to DNS in IP
- Syntex - fcalias name
- Alias can be advertised through Zoneset distribution. Syntex - zoneset distribute vsan 1
- Example:
!
fcalias name ABC vsan 30
member pwwn
exit
!
- Verification: sh fcalias
- Creating zone and Zoneset using ALIAS:
!
zone name ABC vsan 30
member fcalias DISK1
member fcalias DISK2
member fcalias DISK3
member fcalias SERVER
exit
!
!
zoneset name XYZ vsan 30
member ABC
!
!
zoneset activate name XYZ vsan 30
!
+++++++++++++++++++++++++++++++++++
Basic Vs Enhanced Zoning
########################
- By default the full zoneset is local and the active zoneset is fabric-wide
- Order of operations errors can corrupt the Active Zoneset
- Enchanced zoning prevents this by Locking the fabric which ensures people don't
accidentally overwrite each other
- Enhanced Zoning works on per VSAN basis
- The lock on zone is released by "committing" the zoneset
- But if Admin forgets to commit, someone else can release lock as well with
"clear zone lock vsan "
- Enabling Enhanced Zoning
> zone mode enhanced vsan < Per VSAN Basis
> system default zone mode enhanced < For Entire System
- Configuring Enhanced Zoning doesn't change anything for regular FCALIAS and
they still remain local by default
- To solve this problem solution is "Device Aliases"
- Device Aliases serves the same purpose as FC Aliases by binding PWWNs to a user
friendly name but the difference
is that the binding is advertised to the fabric and doesn't remain local only
- Configured as device-alias database
- Changing the zoning mode from normal to enhanced is not disruptive
- Device aliases are advertised through CFS so should not be used in multi vendor environment
+ Configured as "device-alias database"
+ device-alias commit
E.g.>
device-alias database
device-alias name ABC_Server pwwn
device-alias name XYZ_Disk1 pwwn
device-alias name XYZ_Disk2 pwwn
device-alias name XYZ_Disk3 pwwn
device-alias commit
show device-alias database
zone name ABC vsan
member device-alias ABC_Server
member device-alias XYZ_Disk1
member device-alias XYZ_Disk2
zoneset name XYZ vsan
member ABC
exit
zoneset activate name XYZ vsan
sh zoneset pending
zone commit vsan
sh zone active
"sh port-resource module " shows port grouping on MDS
(shared vs dedicaed mode), in 5k all ports are in dedicated mode by default.
HTH...
Deepak Arora
Evil CCIE
