Thursday, September 26, 2013

OSPF Design Issue With Inter Area Routing - Why Is It Not Following Shortest Path ? - RFC 5185



While there are several thoughts in market about how good Design skills can be gained - Where some people have opinion that Design can be only learned from Experience while other may talk about Study as Tool or perhaps availability of Case Study can fasten the process.

On the flip side some Expert Level Certification like CCIE usually doesn't focus on Design side of protocol but rather feature set. For example the Scenario I shown above - It's very easy to trick a CCIE with even this small OSPF setup by asking if they see any design issues with this design. And Most likely the answer is going to be - "No"

Now lets take a look at Design issue with even this small setup:

Lets say R1's Loopback wants to Talk to R5's Loopback. Let's see what Traceroute has to say about path being followed:

R3#traceroute 5.5.5.5 source loopback0

Type escape sequence to abort.
Tracing the route to 5.5.5.5

  1 175.13.0.1 40 msec 100 msec 48 msec

  2 150.15.0.5 160 msec 48 msec 52 msec


Looks good so far as we are talking the shortest path to reach the destination.

Lets review R1's routing table which is next hop in this case:

R1#sh ip route 5.5.5.5
Routing entry for 5.5.5.5/32
  Known via "ospf 1", distance 110, metric 2, type intra area
  Last update from 150.15.0.5 on FastEthernet1/0, 00:01:04 ago
  Routing Descriptor Blocks:
  * 150.15.0.5, from 5.5.5.5, 00:01:04 ago, via FastEthernet1/0

      Route metric is 2, traffic share count is 1

Still being CCIE and looking at the output above, you might not have observed design issue here.

Let's bring the link down between R1-R5:

R5(config)#int f0/0
R5(config-if)#shutdown
R5(config-if)#exit
R5(config)#

*Mar  1 00:39:10.651: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.1 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or detached

And now let's run the Traceroute again to see if we are are following the shortest path as shown in diagram below:




R3#traceroute 5.5.5.5 source loopback0

Type escape sequence to abort.
Tracing the route to 5.5.5.5

  1 175.13.0.1 44 msec 100 msec 12 msec
  2 150.16.0.6 52 msec 48 msec 36 msec
  3 150.26.0.2 72 msec 56 msec 76 msec
  4 150.25.0.5 156 msec 240 msec 116 msec


Dahhhh.....

So we aren't following the shortest path. But rather a far longer/sub-optimal path to reach the destination prefix as shown in diagram below:



So next question that comes in mind is - Why is this happening ? Since rather asking - How do we fix it ? is not the harder part usually for CCIEs :)

Now returning back to first HOW ? part. Let's return back to basics of OSPF route preference which says basically - Prefer O over IA O over E1 over E2 over N1 over N2 routes.

So when Packet comes to R1, it sees 5.5.5.5 as Intra Area route from Area 1 and Inter Area route from Area 0 Prospective.

And per rule - Intra Area route should be preferred over Inter Area route.

Now How do we fix it ?

Hmmm..... Actually Depends on which IOS and Box you are are running.

In newer generation of IOS like IOS-XR, they solved the problem using RFC-5185.

But our old friend IOS still doesn't allow that implementation. So we got to fix it other way which of course involves Downtime if you got to make those changes in production environment.

Let's first revert the configuration to be on same stage we were on earlier:

R5(config)#int f0/0
R5(config-if)#no shutdown
R5(config-if)#exit
R5(config)#
*Mar  1 00:40:37.735: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
*Mar  1 00:40:38.735: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
R5(config)#
*Mar  1 00:40:41.159: %OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.1 on FastEthernet0/0 from LOADING to FULL, Loading Done


R3#traceroute 5.5.5.5 source loopback 0

Type escape sequence to abort.
Tracing the route to 5.5.5.5

  1 175.13.0.1 52 msec 40 msec 68 msec
  2 150.15.0.5 32 msec 64 msec 100 msec


Now Let's apply the fix:

R1(config)#default interface f0/0
Building configuration...

Interface FastEthernet0/0 set to default configuration


R2(config)#default interface f0/0
Building configuration...

Interface FastEthernet0/0 set to default configuration


R1(config)#int f0/0
R1(config-if)#no shutdown
R1(config-if)#exit

R1(config)#int f0/0.12
R1(config-subif)#encapsulation dot1Q 12
R1(config-subif)#ip add 175.12.0.1 255.255.255.0
R1(config-subif)#exit

R1(config)#int f0/0.112
R1(config-subif)#encapsulation dot1Q 112
R1(config-subif)#ip add 150.12.0.1 255.255.255.0
R1(config-subif)#exit

R1(config)#router ospf 1
R1(config-router)#network 150.12.0.1 0.0.0.0 area 1
R1(config-router)#exit

R2(config)#int f0/0
R2(config-if)#no shutdown
R2(config-if)#exit

R2(config)#int f0/0.12
R2(config-subif)#encapsulation dot1Q 12
R2(config-subif)#ip add 175.12.0.2 255.255.255.0
R2(config-subif)#exit

R2(config)#int f0/0.112
R2(config-subif)#encapsulation dot1Q 112
R2(config-subif)#ip add 150.12.0.2 255.255.255.0
R2(config-subif)#exit

R2(config)#router ospf 1
R2(config-router)#network 150.12.0.2 0.0.0.0 area 1
R2(config-router)#exit

R1#sh ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2           1   FULL/BDR        00:00:36    175.12.0.2    FastEthernet0/0.12
3.3.3.3           1   FULL/BDR        00:00:33    175.13.0.3      FastEthernet0/1
2.2.2.2           1   FULL/BDR        00:00:33    150.12.0.2   FastEthernet0/0.112
6.6.6.6           1   FULL/BDR        00:00:34    150.16.0.6      FastEthernet2/0
5.5.5.5           1   FULL/BDR        00:00:33    150.15.0.5      FastEthernet1/0


Let's perform the testing one final time to see if this fixes the design issue:


R5(config)#int f0/0
R5(config-if)#shutdown
R5(config-if)#exit

R3#traceroute 5.5.5.5 source loopback 0

Type escape sequence to abort.
Tracing the route to 5.5.5.5

  1 175.13.0.1 40 msec 40 msec 48 msec
  2 150.12.0.2 108 msec 1424 msec 88 msec
  3 150.25.0.5 216 msec 96 msec 144 msec

So we are good now :)

Below are the initial configuration in case you want to test this out by your own:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
                                       Initials
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

R1
++

!
en
!
conf t
!
no cdp run
!
no ip do lo
!
ho R1
!
line con 0
 logg syn
 no exec-time
 exit
!
logging buff
!
logging buff 7
!
logging buff 5000
!
int lo0
 ip add 1.1.1.1 255.255.255.255
 exit
!
int f0/0
 ip add 175.12.0.1 255.255.255.0
 no sh
 exit
!
int f0/1
 ip add 175.13.0.1 255.255.255.0
 no sh
 exit
!
int f1/0
 ip add 150.15.0.1 255.255.255.0
 no sh
 exit
!
int f2/0
 ip add 150.16.0.1 255.255.255.0
 no sh
 exit
!
router ospf 1
 net 1.1.1.1 0.0.0.0 a 0
 net 175.12.0.1 0.0.0.0 a 0
 net 175.13.0.1 0.0.0.0 a 0
 net 150.15.0.1 0.0.0.0 a 1
 net 150.16.0.1 0.0.0.0 a 1
 exit
!
end
!

#################################


R2
++

!
en
!
conf t
!
no cdp run
!
no ip do lo
!
ho R2
!
line con 0
 logg syn
 no exec-time
 exit
!
logging buff
!
logging buff 7
!
logging buff 5000
!
int lo0
 ip add 2.2.2.2 255.255.255.255
 exit
!
int f0/0
 ip add 175.12.0.2 255.255.255.0
 no sh
 exit
!
int f0/1
 ip add 175.24.0.2 255.255.255.0
 no sh
 exit
!
int f1/0
 ip add 150.26.0.2 255.255.255.0
 no sh
 exit
!
int f2/0
 ip add 150.25.0.2 255.255.255.0
 no sh
 exit
!
router ospf 1
 net 2.2.2.2 0.0.0.0 a 0
 net 175.12.0.2 0.0.0.0 a 0
 net 175.24.0.2 0.0.0.0 a 0
 net 150.26.0.2 0.0.0.0 a 1
 net 150.25.0.2 0.0.0.0 a 1
 exit
!
end
!

#################################


R3
++

!
en
!
conf t
!
no cdp run
!
no ip do lo
!
ho R3
!
line con 0
 logg syn
 no exec-time
 exit
!
logging buff
!
logging buff 7
!
logging buff 5000
!
int lo0
 ip add 3.3.3.3 255.255.255.255
 exit
!
int f0/0
 ip add 175.13.0.3 255.255.255.0
 no sh
 exit
!
router ospf 1
 net 3.3.3.3 0.0.0.0 a 0
 net 175.13.0.3 0.0.0.0 a 0
 exit
!
end
!


#################################


R4
++

!
en
!
conf t
!
no cdp run
!
no ip do lo
!
ho R4
!
line con 0
 logg syn
 no exec-time
 exit
!
logging buff
!
logging buff 7
!
logging buff 5000
!
int lo0
 ip add 4.4.4.4 255.255.255.255
 exit
!
int f0/0
 ip add 175.24.0.4 255.255.255.0
 no sh
 exit
!
router ospf 1
 net 4.4.4.4 0.0.0.0 a 0
 net 175.24.0.4 0.0.0.0 a 0
 exit
!
end
!

#################################


R5
++

!
en
!
conf t
!
no cdp run
!
no ip do lo
!
ho R5
!
line con 0
 logg syn
 no exec-time
 exit
!
logging buff
!
logging buff 7
!
logging buff 5000
!
int lo0
 ip add 5.5.5.5 255.255.255.255
 exit
!
int f0/0
 ip add 150.15.0.5 255.255.255.0
 no sh
 exit
!
int f0/1
 ip add 150.25.0.5 255.255.255.0
 no sh
 exit
!
router ospf 1
 net 5.5.5.5 0.0.0.0 a 1
 net 150.15.0.5 0.0.0.0 a 1
 net 150.25.0.5 0.0.0.0 a 1
 exit
!
end
!


#################################


R6
++

!
en
!
conf t
!
no cdp run
!
no ip do lo
!
ho R6
!
line con 0
 logg syn
 no exec-time
 exit
!
logging buff
!
logging buff 7
!
logging buff 5000
!
int lo0
 ip add 6.6.6.6 255.255.255.255
 exit
!
int f0/0
 ip add 150.26.0.6 255.255.255.0
 no sh
 exit
!
int f0/1
 ip add 150.16.0.6 255.255.255.0
 no sh
 exit
!
router ospf 1
 net 6.6.6.6 0.0.0.0 a 1
 net 150.26.0.6 0.0.0.0 a 1
 net 150.16.0.6 0.0.0.0 a 1
 exit
!
end
!

HTH...
Deepak Arora
Evil CCIE



Friday, September 20, 2013

Network Designer/Architect Vs Network Engineer...Theory Vs Real Life - Part 1


As I am starting this series today, back in my mind I am keep thinking that this series is going to be end up being a story book. Prime reason seems to be that in this series I am not going to talk much about Technologies and even if I does, most likely the Non-Technical side of it.

Don't expect me to show you some incredible design skills at the moment since there are some better guys to talk about those things in market. Russ White, Petr L, Ivan P, Brian M and of Course Scott Morris are rather better people and each of these keep writing about that stuff on different places at web.

This series is based on rather my Experience with Network Designing & Architecture [ A little Experience I have :) ], Few Designer/ Architect interviews I attended, Network Industry's expectations from CCIE guys as Network Designers, List of things that I feel affects Network Designs in real life, CCIE/CCDE market in India Vs US including Job Market, What are my opinions about CCDE [ Though Still I am targeting to get it :) ] And last but not least my Technical blog posts as I progress with my CCDE Study though I got to get my CCIE DC first, But I am still trying to manage couple of hours every week to go through some design stuff :)

Every Criticism & feedback is going to be highly appreciated. A copy of this series posts I am going to post over Ethan Banks CCDE Study group (https://groups.google.com/forum/#!forum/ccdegroupstudy) until he blocks me by marking as SPAM :). Though the group seems to be little slow at the moment in terms of sharing ideas or picking up case studies and discussing design/decision side of it. But I am hoping it will pick up the pace sooner or later.

Again if you are looking for pure design stuff at the moment - You better read writeup from people I mentioned above.

I'll talk about those things in future but not at the moment. I got to understand the psychology of Network Designer/Architect first or perhaps to get to level where I can understand them.

Stay Tuned...

HTH...
Deepak Arora
Evil CCIE

Tuesday, September 17, 2013

Free CCNA Data Center Video Training - @ Cisco Learning Network

In case you guys haven't noticed this. Cisco Learning Network is offering CCNA Data Center Video Training in free. Most of the sessions were ran by Robert Burns which is CCIE Data Center. Below are access details:







https://learningnetwork.cisco.com/docs/DOC-17214

https://learningnetwork.cisco.com/community/learning_center/recorded_tech_seminars

HTH...
Deepak Arora
Evil CCIE

Tuesday, September 10, 2013

Cisco Catalyst 6500/6800 Instant Access - Answer To Cisco Nexus 2000 AKA FEX For Enterprise ?

As Cisco Launched its Next Gen Catalyst 6000 Series as Catalyst 6800 based on One of it's most successful Enterprise switch product Cat 6500's DNA at recent Cisco Live Event At Orlando, There was an advantage that competing Nexus Series had even as solution to Enterprise market was Nexus 2000 AKA FEX, which was offering lots of benefits.

While the roadmap/investment protection of Nexus Family is yet to be clarified by Cisco as they keep coming with newer versions and flavors, the Cat 6500 always seen as very mature platform for Enterprise solutions.

Even to compete further with Nexus family, Cisco Catalyst team came up with a latest solution recently as INSTANT ACCESS (IA).

The benefits Cisco IA offers are some what similar to Nexus 2000/FEX. Though there are few hard requirements like :

> SUP 2T
> VSS CODE
> Switches To be configured in VSS Mode even if it's a standalone chassis
> 6904 10/40Gig line cards to be operated in 10Gig mode only

Further Readings:









HTH...
Deepak Arora
Evil CCIE

Wednesday, August 14, 2013

IP Accounting - A Deeper Look From QOS Prospective

Got a call from one of a Friend (Network Engineer) yesterday looking for a quick way to verify QOS marking for a specific traffic.

Now there are multiple ways to get this done. One of the easiest way is to use an ACL matching specific IP Precedence values and apply it on interface in incoming direction and see if you are getting hits. Though it's a quick way but there are some limitations as well. For example - Say I am expecting incoming traffic to carry IP Precedence 5 value. And if you are matching IP Precedence 5 in ACL, you should see hits on ACL. On the flip side if for some reason traffic lost it's marking some where in middle. You won't see hits. Which is what usually we need to know while troubleshooting. But on the flip side you still couldn't determine what was actual IP Precedence value that traffic was carrying at that point which didn't create hit on the ACL. 

Now you might be thinking, that's easy. We can create one ACL per IP Precedence value (Total 8) and we should be all good. Which is Okay. But lengthy way. Specially if you are little lazy at Typing like me :)

There is an alternate way as well. All most of us have used "IP Accouting" feature at some point of our Networking Career. But this feature has couple of more branches which are not explored that much per say. One of function it provides is something that can help us with above mentioned situation and even easier and faster in terms of configuration and verification.

Let's review our topology:


Say on R3, I am expecting Server to send traffic towards R3's Loopback with IP Precedence Value 5. But on the flip side I am really not 100% sure if that is really happening.

Now we can create 8 ACLs matching IP Precedence Values 0-7 and apply that ACL on F0/0 of R3 in incoming direction. And than later look at "sh access-list " command output to see which one out of eight ACL entries actually got hit to figure out which IP Precedence value traffic was actually carrying.

Nahhh..... Let's look at other way

Lets enable the feature on R3's F0/0 interface for incoming packets. Though this feature is supported in outbound direction as well.


R3#sh run int f0/0
Building configuration...

Current configuration : 125 bytes
!
interface FastEthernet0/0
 ip address 23.0.0.3 255.255.255.0
 ip accounting precedence input
 duplex auto
 speed auto
end


Lets send couple of pings to see our traffic is carrying which IP Precedence value:


SERVER#ping 3.3.3.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 352/428/524 ms

 
 Lets verify back on R3

R3#sh int f0/0 precedence
FastEthernet0/0
  Input
    Precedence 0:  5 packets, 570 bytes


As you can see we received the traffic on R3 marked with IP Precedence 0, Which means default value.

Hmmm... Seems like our QOS policy is not working.

Lets set up one and see if that changes the situation as per our requirement.

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#class-map match-all ICMP
R2(config-cmap)#match protocol icmp
R2(config-cmap)#exit
R2(config)#
R2(config)#policy-map PREC5
R2(config-pmap)#class ICMP
R2(config-pmap-c)#set ip precedence 5
R2(config-pmap-c)#exit
R2(config-pmap)#
R2(config-pmap)#int f0/1
R2(config-if)#service-policy output PREC5
R2(config-if)#end


Now as configuration seems to be in place to get our job done. Let's re-send the Pings from Server and see if they are carrying IP Precedence 5 now.


SERVER#ping 3.3.3.3

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 232/488/688 ms


Let's re-verify on R3

R3#sh int f0/0 precedence
FastEthernet0/0
  Input
    Precedence 0:  10 packets, 1140 bytes
    Precedence 5:  5 packets, 570 bytes


Hmmm....The policy seems to be working now. But old matches are still there and seems to be increased. Which looks confusing.

Okay, lets clear the mess and re-run the test and verify one last time.

R3#clear counters f0/0
Clear "show interface" counters on this interface [confirm]


R3#sh int f0/0 precedence
FastEthernet0/0
  Input
    (none)



Now counters are cleared. Let's Re-Test


SERVER#ping 3.3.3.3 r 10

Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 200/346/508 ms


Final Verification:

R3#sh int f0/0 precedence
FastEthernet0/0
  Input
    Precedence 5:  10 packets, 1140 bytes


 Nice... Isn't It ? :)

Further Readings

http://www.ciscopress.com/articles/article.asp?p=764234&seqNum=4

HTH...
Deepak Arora
Evil CCIE