Layer 2 Traceroute - Another Cisco Baby To Help Operations Guys

Well it's been a while or probably years ago I wanted to talk and write about this interesting Cisco baby called " Layer 2 Traceroute ". I called it Cisco Baby because it requires Cisco Discovery Protocol (CDP) to be enabled in order to function.

Now Idea is simple, In traditional L2 Campus or Enterprise Network our Network Operations Team keep getting several requests such us tracing user port and change it's vlan to something else or maybe a troubleshooting call where we need to identify source port of user or server etc. The traditional approach usually was to first hop on to switch hosting SVI for given VLAN , Figure out Layer 2 address for given IP (User/Server IP) and do reverse engineering by tracing that Layer 2 Mac Address hop by hop until we hit the user port configured in access vlan or Server that might be using trunking in a virtualized network.

Here is a quick example of traditional approach:

Now Here is how this wonderful IOS tool helps you trace quickly

Further Readings:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/configuration/guide/book/l2trace.html

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3550/software/release/12-1_13_ea1/command/reference/3550cr/cli3.pdf

http://packetpushers.net/tracing-a-layer-2-path-on-cisco-nexus-switches/


HTH...
Deepak Arora
Evil CCIE

OSPFv3 Gets Family

It's been a while since I touched Cisco CLI during Non Office Hrs. There are lots of things keeping me busy enough these days like I have been doing a Proxy relocation to DC project which means tons of stuff to deal with as DC is a fairly complex structure with lots of moving pieces like Load Balancer ( SLB, GSLB, LLB), Firewalls and above that this particular DC runs all Juniper Devices for Routing , Switching & Security stuff.

On the flip side I am thinking to seriously start working on CCDE from Mid May 2015 onwards with first lab attempt probably in Feb or May 2016. Though initially CCIE Wireless was on Radar for this and coming year but as Cisco has just announced new blueprint , the idea went on to backseat for a while now. Voice could have been interesting also but similarly CCNA and CCNP Collab tracks are going to replace existing NA and NP Voice track soon and moreover I haven't been doing much around Voice at work as well so should consider it some time next year.

Now back on to idea of today's post about OSPF Address Family Structure. Well it's been a while since Network Industry was talking about one of advantage of IS-IS over OSPF which is IS-IS supports IPv6 natively. For IS-IS it was just a matter of adding new TLV (Type Length Value) to existing IS-IS packet ( TLV 232 & 236 ? I guess). Where as with OSPF it was a different story. They actually had to write a new routing protocol right from scratch now known as OSPFv3. Which introduced another problem from network design and management standpoint that if we just want to run OSPF as choice of routing protocol for both IPv4 & IPv6 on network as administrator, we got to run two separate routing protocol instances as OSPFv2 for IPv4 and OSPFv3 for IPv6. Which means two separate routing topologies.

Now with OSPFv3 Address family (AF) structure support we get rid of this problem of running two separate routing instances somewhat. Why "somewhat" and not "completely" ?

Well while this AF structure solves the multiple instances requirement problem, it has it's own limitations like:

> Specific IOS version like 15.x required, which may or may not be supported across all existing routers & switches serving as Distribution block

> The AFv4 requires IPv6 Link Local address on all IPv4 interfaces in order to function, otherwise it doesn't allow you to bind OSPFv3 AFv4 instance on interface. The Link local addresses are used as source address of packet over interfaces connecting OSPF enabled devices

> The device with capability of AFv4 still couldn't talk to traditional OSPFv2 Device since there is a Address Family bit carried inside new Hello packets which traditional OSPFv2 device don't understand

Which means the deployment is more likely to be successful only if it's a Green Field deployment.

Here is a quick lab with quick results :)

Further Readings:

http://www.ciscopress.com/articles/article.asp?p=2294214&seqNum=4

http://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/open-shortest-path-first-ospf/whitepaper_c11-668030.html

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_ospf/configuration/xe-3s/iro-xe-3s-book/ip6-route-ospfv3-add-fam-xe.html#GUID-10C6CAB4-C220-47A5-9DEA-F57319BCFFDA

HTH...

Deepak Arora

Evil CCIE

OSPF Path Selection With Default Information Originate - How You Gonna Fix It ? - Part 2

As OSPF recent design issue discussed here can possibly turn into obnoxious CCIEv5 R&S troubleshooting scenario, I got one email from another friend who also ran into interesting OSPF design issue recently around OSPF default information originate too.

Here is a quick topology of his network design:

At high level the design looks pretty simple. R1 & R2 are peering with WAN cloud using E-BGP. Between R1-R2-R3-R4-R5 we are running OSPF Area 0. To ensure R3-R4-R5 have reachability to external prefixes (Since it's always good to avoid BGP to IGP Redistribution), R1 & R2 are injecting default route into the network using "default-information originate always metric-type 1" option.

Also R1 & R2 are connected to each other over a link running OSPF.

Now let's talk about problem...which is the fun part :)

If R1's WAN connection goes down, R4 can't reach any external prefixes using default route any longer. Whereas at the same time both R3 & R5 don't lose their connectivity to same external prefixes.

Now ideally once R1 find it's WAN interface down, it should get default route from R2 and should be able to route R4's traffic in that direction.

Let me show you same behaviour using CLI :

Before R1's WAN Link goes down

After R1's WAN Link goes down 

No ICMP Packets Received Any Longer on R6

R1 is not installing default route from R2

See if you can figure out " WHY " behind this and offer different ways we can fix this design issue. Feel Free to post you solution, recommendation and findings under comments section. I'll update the post around next weekend with my findings and solutions to the design.

HTH...

Deepak Arora

Evil CCIE



Update

 In SPF Debug

OSPF Path Selection With Default Information Originate - How You Gonna Fix It ? - Part 1

I recently got a call from friend , he was stuck with an interesting OSPF Path Selection issue. Obviously the things didn't work the way he expected it to be :)

Now initially he was been given a Consulting task that involved Design & Configuration changes to meet a new network requirement.

Here is how the network initially looked like :

The customer had active-active WAN connections where these connections were also serving both WAN as well as Internet connectivity. Where as in LAN side they were using OSPF as shown in Topology above.

Now the new requirements given were:

> Brought Up New Internet Connection that will act as Primary Path to reach internet.

> In case the New Internet connection goes down, the old WAN cloud should still work as backup path to reach internet

> From R4's perspective, the path towards R2 should be primary path to reach internet (Through New Internet GW) while path through R3 should be backup path. I am not sure why this was a Hard Requirement but I guess the path from R4 to R3 was through RF link as shown below:

> Introduction of Static Routes, PBR, NAT for whatever reasons were not allowed to be used to solve this problem 

Now my friend tried to solve this problem in an interesting way. He Introduced RIPv2 as new IGP between R5-R3-R2 to ensure he doesn't have to change anything into existing IGP since there was a large IGP domain connected to this setup.

Using RIPv2 he injected default route into RIB of R2 and R3. He had to change distance of RIPv2 originated default route so that it can get priority over BGP injected default route. Than he tried Injecting Default Routes into OSPF domain towards R4 using "Default Information Originate " option. Since the control over path selection was required here, During default route injection he used Metric Type as 1 with Higher metric given while injection route at R3 as shown below:

Obviously the solution didn't work as expected and here is why:

As you can on R4 we still have two routes instead of one in Routing Table.

Can you figure out what's the problem here and how we can fix it in easiest way ?

Or You Have any other better solution to fix it while meeting the given restrictions ?

I'll post my solution to the problem after 2 weeks.

To be on same page here is the Addressing I Used in while labbing this up:

Here is relevant configuration:

 

Good Luck  !!! :)

HTH...

Deepak Arora

Evil CCIE

Update For Ref: See comments section for further information.

HTH...

Deepak Arora

Evil CCIE

Does EIGRP Feasible Successor Always Work As Successor Fails ?

A recent discussion with friends brought this idea in my mind to write about this exiting subject. While I won't say all but many among them thought that it's always good to have EIGRP Feasible successors into Network while designing an EIGRP based network (Which of course is true ). And if EIGRP successor Route Ever fails, the EIGRP feasible successor will be installed quickly as it's a second best route in EIGRP Topology table based on feasibility condition, which will help minimizing convergence time.

And this is where most people start assuming that it's always going to be the case. Which of course is not true :)

Let's test this quickly based on following topology:

From R1's perspective it has three different paths to reach the destination - 5.5.5.5

Now let's review R1's routing table to first find which path is preferred.

As we can see, the middle path has been chosen as best path based on Dual Algorithm. Now let's next review R1's EIGRP topology table to figure out if we have feasible successor chosen at all and if So than which path.

 

 As we can see, the path through R4 has been chosen as second best path (Feasible Successor). Since there is no other entry in Topology table showing path through R2, which means it's neither successor nor feasible successor and has failed feasibility condition.

Let's verify Feasible Distance (FD) and Reported Distance (RD) for path through R2 by shutting down the path through R3 and R4.

Now here is an interesting scenario:

> Path through R3 is the best path (Lowest Metric)

> Path through R4 is meeting feasibility condition (making it feasible successor). But overall cost to destination is worst if we compare all three paths metric.

> Path through R2 is actually second best path based on total metric but got out of the equation as it failed feasibility condition

Though we can see that topology doesn't include any potential link which can lead traffic back to original source while forwarding traffic towards destination 5.5.5.5 , but EIGRP fails to recognize this fact. 

This is where we find this true that eventually EIGRP is an Advance Distance Vector protocol as it tries to avoid any possible looping with help of Dual Algorithm but is not always successful to find it's goal. But not as good as a Link State Routing protocol which would have the complete picture of the topology.

Now in this scenario what you think would happen if Successor Route fails ?

If we go by theory discussed earlier in the post, EIGRP feasible successor should take over. Right ?

But that would mean sub-optimal routing. 

But don't worry. EIGRP is intelligent enough still and it finds Optimal Path here based on overall cost to destination and avoiding sub-optimal path.

 
 To my surprise during this test two commands didn't work the way I expected. Which I must figure out sometime :)

Further Readings:

http://www.ciscopress.com/articles/article.asp?p=1763921&seqNum=5

http://books.google.co.in/books?id=XkM6vxsVJEsC&pg=PA78&lpg=PA78&dq=eigrp+convergence+with+feasible+successor&source=bl&ots=mvVF2sg2_K&sig=1jz3SMiK6TiRNnSdj1LKTIj8AHk&hl=en&sa=X&ei=EIxWVMriI8ekuQS-gIL4AQ&ved=0CD8Q6AEwBQ#v=onepage&q=eigrp%20convergence%20with%20feasible%20successor&f=false

http://www.cisco.com/c/en/us/support/docs/ip/enhanced-interior-gateway-routing-protocol-eigrp/16406-eigrp-toc.html#feasibleandreported

http://packetlife.net/blog/2010/aug/9/eigrp-feasible-successor-routes/

http://rovingengineer.wordpress.com/2010/07/28/eigrp-feasible-successor-routes/


HTH...
Deepak Arora
Evil CCIE

OSPF Route Preference: You Sure You Know It ? ... RFC 3101...Part 2 (Final)

It's sunday, got some time today. Let's wrap up this series quickly.

The physical and logical topology is gonna remain same. The only thing I changed in this part is ... IOS Version :)

 

R3 is Still the Translator:
 

Let's review the routing table of R2 quickly again and see how things are:

So now the question is why results are different with different IOS ? ( IOS used in part 1 was 12.4 (20) T ).

The previous version was actually following RFC 1587 in which the route preference is same as mentioned in post 1 i.e. E2 is preferred over N2.

But the later IOS followed the RFC3101 implementation. Which states that LSA with P-bit set is preferred over the one which has P bit as Zero. The P bit is set to 1 by ASBR while redistributing the routes into NSSA except the situation where ASBR is also ABR and P bit remains unset (0).

But as an Engineer if you would like you can still go back to older IOS behavior as following:

The reason in Part 1 , the traffic (Data Plane) took the direct path because of same cost to Forwarding Address.

Further Readings:

http://blog.ipspace.net/2008/01/e1-and-e2-routes-in-ospf.html

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_ospf/configuration/15-e/iro-15-e-book/iro-ospfv2-nssa-cfg.html#GUID-C335FD95-ED21-4A51-894F-8B63119186EC

http://networklessons.com/ospf/ospf-path-selection-explained/#ixzz3GPsDJ8N8

HTH...
Deepak Arora
Evil CCIE