Sunday, December 20, 2020

BGP Nuts - The Crazy Metric Story



Recently ran into an interesting BGP behavior. In order to convey the issue, I have oversimplified the design.

R2 (AS 100) has a network host 22.22.22.22/32 which is trying to reach R7 (AS 200) attached network host 77.77.77.77/32. 

As you can see, between AS 100 & AS 200 we have multiple exit points for the purpose of redundancy. However the interesting twist here is that those BGP next hops are learned from different IGPs.

54.0.0.5 <- Learned from EIGRP
63.0.0.6 <- Learned from OSPF

Given the scenario, what you think about which BGP route and next hop would make into the routing table ?

Well...most likely the answer would be BGP next hop learned from EIGRP because (Assuming... 1. " no auto-summary " is now default for EIGRP (resulting into same Prefix Length for route) , 2. EIGRP AD is 90 compare to OSPF Intra Area route which has AD 110. So based on theory how Router Process this information, the EIGRP learned Next hop (54.0.0.5) should be the one picked up by BGP Best Path Selection Algorithm and should be picked up as the best route over OSPF learned Next Hop (63.0.0.6)

All good ? ... Let's look at the BGP RIB to find it out and How IGP next hops are processed based on standard BGP Next Hop Processing approach that we know from standard BGP theory. Also let's run a quick traceroute towards 77.77.77.77 sourcing from 22.22.22.22 on R2



Now sure if you expected that...right ? :)

From BGP RIB standpoint we can clearly see that BGP route that's making into routing table & FIB essentially is the one that has OSPF learned BGP next hop.

Here is what our OSPF database looks like



Let's do a basic failover test by shutting down interface g6/0 on R3 




So things works fine and as expected here, so let's "un-shut" the interface in order to dig deeper into the behavior ( a little unexpected one)




So why this behavior ?...Well this is where BGP plays a little dumb. It actually ends up comparing METRICs across two IGPs since it doesn't have view of EIGRP Topology Table and OSPF Link State Database. Now if you go back and check IGP metrics for next hops learned from EIGRP vs. OSPF, The metric for EIGRP route is pretty high (in numeric ) as 3072 vs. OSPF route as 2 (Refer to first CLI screenshot in the post)

Now what if we make the OSPF metric look bigger than EIGRP by manually increasing OSPF cost to see if that would allow R2 to install EIGRP learned route as best route





And Yes it works as expected this time too :)

Now for fun let's make the metrics look same by again changing the OSPF cost manually


Well that works too as expected. Now interesting question here is - Since IGP metrics learned from OSPF and EIGRP look same, can we turn ON " i-BGP Multipath " and achieve load balancing.

Let's try that 


Well that works well too and as expected.

One of the problem incase you haven't noticed yet with this design is - Assume the path through OSPF has many more hops added before you reach to the OSPF learned BGP ASN exit point. Since OSPF cost metrics are much lower value compare to EIGRP, You will end up taking sub-optimal path from R2's standpoint in this case which may not be desired behavior. Alternatively You can replace EIGRP with IS-IS and you will by default still end up following OSPF learned BGP next since in most IOS version 10 is the default IS-IS metric for each interface hop. Obviously in practice you should try to run Single IGP across AS 100 here to avoid such issues though M&A scearios are always interesting and challenging. :)

Try this with " DMZ Link-BW " for Unequal Cost Multi-Path and I am pretty sure it will be fun. BGP AIGP NLRI is another interesting bit if you care depending upon the design. 

And of course, the above mentioned logic doesn't apply to following design where we have single exit point being reachable through both IGPs




To learn some more around BGP anomalies and somewhat un-predictable behavior:



And if You want to master BGP from Design Standpoint, I would highly recommend " BGP Zero to Hero Design Masterclass " from my friend Orhan Ergun



HTH...
A Network Artist

Thursday, May 28, 2020

The Mythical "Networking Expert" Unicorn - Part 1



" Expertise is a Myth " - Yes you read that right !!!

Recently there were couple of different posts/threads going on over Linkedin where people were asking a very common question that most people in Networking/Technology industry struggle to answer and in most cases it only causes the them anxiety - " Since there are just too many new Technologies & Solutions those have surfaced in last couple of years, how do I stay relevant moving forward and what should I focus on? "

While the easiest answer you always hear from people would be " It depends" and they are probably right (Let's keep it for another post as it's not really as hard as a problem if you apply some science to it, "Yes there are proven methods" and you don't necessarily need to completely depend upon "what your gut tells you" which is what I guess 99% in real world would people rely upon, if you ever doubt that - Please be my guest to show me how you plan your career decisions in terms of a structured approach you follow to make those decisions).

Perhaps the 5 Ws is a good place to start.

Now on the same linkedin thread, many people suggested to follow T-Shaped skill set which essentially says - Be an Expert in few selective areas while continue to me a generalist in rest of areas.

Sounds plain, simple & logical - isn't it ?

But that leads us to another interesting question which is "How do I know if I am an Expert in a certain area?" - Since in general no one tells us how to gauge our expertise or perhaps the real question is "Does Expertise exists for real ?"

Here is what Wikipedia says -  "An expert is somebody who has a broad and deep competence in terms of knowledge, skill and experience through practice and education in a particular field."


Now as you can see, while those are some good statements and sensible ones too, they still don't tells us ways to measure expertise correctly. So still what we are struggling with is quantification of our Expertise persona.

Now obviously it just can't be unsolvable problem. But answer essentially lies in "How far you want to go on a quest to find these answers?" since it has its roots in Psychology, Critical thinking and many other places including:

- What's you motivation to be seen as an Expert ? Ever thought about it?
   In general it has it's roots spread across - Emotional Reasons, Psychological
   reasons, Social reasons, Financial reasons and so forth.

- Are you really trying to solve the real problem or just trying to address
   that in the realm of your own area of experience/expertise ?



- And in real world in order to do what I just mentioned above, you got 
   to step out of your comfort zone, So are you willing to do that?

- Since every environment has its own set of nuances involved including 
   its own set of constraints, one solution won't meet everyone's needs.

- Are you good at interacting with people or you are just comfortable 
   talking to Network devices and consider yourself as an introvert? 

- Are you keen to learn but more importantly put in practice lot of those 
   non-technical but essential skills which you would actually need to first  
    understand the problem landscape correctly?

- Are you keen to do actual hard-work to gather data points and validate 
   your understanding and hypothesis on a given topic or problem a
   statement? - Trust me, this is where 99% people usually fail as in most 
   cases they have lot of hypothesis and solutions which only they think 
   makes sense with no validation and Data to prove their point. And in
   many cases the data in question is too generic.

- Are you keen to constantly challenge yourself for good? (Since at times 
   it's a restless & never ending process - Though many enjoy that 
   pain being nerd)

- How do you find balance between Life, Work and Passion (Do you have 
   a method/approach to apply?)

- Do you have plans or just aspirations? (In my personal view aspirations 
   are 1000 kg a $ and plans are expensive)

- Are you another victim of Dunning Kruger effect or Big fish little pond 
   effect ?

- Are you in love with Problem or in love with Solutions? Unless a problem
    is very well defined, it can't really be solved or else your solution in 
    most cases would be just another duct tape which will fix something        
    temporarily but increase complexity by large margin.

- Are you open to listen and learn from other or you are the smartest 
   person in the room that knows the answers always? The most complex
   set of problems are solved best collectively rather than by just throwing
   your point of views into the equation.

L&D Cartoon – Too Busy To Learn | L&D Generous


- And last but not least - Are you willing to fail many times before you 
  succeed or even partially succeed? 


So as long as you have answers to those questions, that's a pretty good place to get started. :) , and if not - Keep continuing to play giant pretending game by projecting yourself as an Expert. :) ... since many people still love illusions or perhaps enjoy living under the rock.






Further stuff to go through: (Make sure you watch them few times)



The Technology Fallacy | The MIT Press


HTH...
A Network Artist

Thursday, April 23, 2020

Do You Really Need VxLAN in a Modern Data Center ?




It started as a thread on Linkedin yesterday when someone posted a sort of glimpse of his free Cisco ACI training on linkedin. Later a Network Architect joined the conversation thread by asking the Original poster "Would you prefer ACI or BGP EVPN VxLAN based standard fabric ?". Of course the Architect suggested he would prefer BGP EVPN VxLAN fabric (That's mouthful man :) )

As usual out of my curiosity (without being a DCN expert - Yes I only worked on 3 ACI projects and Zero Cisco Nexus 9K BGP EVPN VxLAN fabric (VTS) so far), I asked this self proclaimed Network Architect another question...very basic in my view:

In modern Data Center context, Where in VxLAN actually plays an important role. Or is it just another snowflake in Networking realm in order to keep your App guys happy since you are not good at Saying " NO " and push them to build better Apps those works well on L3 vs. Sure guys, I can help you with L2 stretches for your App Clusters and for those totally broken old legacy Apps which only works on L2 best.

The point being:

  1. Most modern App works well on L3 in conjunction with DNS
  2. From Cloud Native Apps perspective, L2 shouldn't happen to begin with
  3. vMotion works well on L3 since 6.x release
  4. Inter DC Cluster using L2 are well known bad design practices (Some people tried it with even firewalls and failed miserably)
  5. At least in my personal view, Most Enterprises still run MLAG or vPC thingy for server redundancy compare to perfect world (Original Hyperscaler ideas) where you either run Routing with Servers and VxLAN boundary starts from Hypervisor layer to avoid broken MLAG and STP implementations
  6. Let's agree on L3 based transport is much better compare to L2
  7. Perhaps we have seen enough DC meltdowns in past due to broken L2 and broadcast loops by now ?
  8. Did you ever notice that most Hyperscalers DCs don't run VxLAN in their DC fabrics (Pick your favorite one) - https://engineering.linkedin.com/blog/2016/03/project-altair--the-evolution-of-linkedins-data-center-network
  9. Perhaps there are good reasons someone like AWS and AZURE don't let you run VxLAN between your On Prem DC and Virtual DC they host (VMware on AWS is a corner case and rather a joint GTM thingy)
  10. How many of your Use Case requirements are persistent vs. temp in nature ( Migration and Maintenance were always temp requirements in my view at least)
  11. What value Bridging over L3 really has today to begin with
  12. Shouldn't we rather focus on solving real problems instead of pretending 
  13. Don't forget you can't have unlimited VxLANs in real world like most vendor slides tell you otherwise
  14. Focus on your failure domains/blast radius and create a strategy to mitigate that - Which doesn't seems easy in VxLAN environment and more so in Controller based Clos Fabric (Single Plainer) running VxLAN
  15. And since Overlays such as VxLAN seems to be the new norm in Enterprise DCs, Make sure you are familiar with some of well known Overlay Problems besides challenges such as Visibility, Security and so forth
  16. Make sure you fully understand the VxLAN convergence process with EVPN added while understanding your dependencies on Physical Topology, Underlay Convergence and other nuances
  17. Make sure you understand Leaky Abstractions & Grey Failures very well in context of VxLAN deployment
  18. Understand some myths about Clos Fabric that your favorite vendor otherwise won't tell you - Part 1, Part 2, Part 3 & Fact Check

And of course I am totally ignoring:

  • Level of complexity VxLAN and EVPN brings into Design...You either got to read through few books to become an SME or your must be Genius
  • NSX over ACI never took off ... sort of
  • Most VxLAN implementations are still vendor specific and ACI runs modified VxLAN header
  • VxLAN & EVPN across multi vendors in Single environment... Good luck (Though things have improved a bit over time)
  • MPLS & SR in DCN Fabric...Perhaps needs a separate discussion
  • I am not aware of any strong one to one mapping in context of Overlay and Underlay correlation for simplified Day 2 and operations
  • VxLAN vs Geneve (What NSX-T uses now) vs STT vs NvGRE (Assuming you love MS) vs Dove....Needs separate discussion
  • This is not a scalability discussion but more around to find real world use cases for VxLAN
  • MS Fabric today (Not Originally) does run VxLAN since 2015. Spoke to few MS friends, seems like Bing was Broken from APP designs perspective so they had to do it in order to Keep Business and App guys happy....BTW who runs Bing anyway :)
  • If you are running HW VTEP with MLAG for servers. Perhaps you got to relook into your design and define what purpose VxLAN really serves
  • We still got plenty of other HW and SW in DC context that don't understand VxLAN and more importantly have no HW acceleration
  • Role of VxLAN offload on NIC, DPDK, SR-IOV and other stuff....perhaps needs a separate discussion
  • HW and SW VTEP Integration and how broken those are...let's keep it for another day
  • IPv6 DC/IPv6 Only DC and VxLAN v6... Perhaps need another thread
  • Control Plane, Data Plane & Policy Plane Security in Context of VxLAN - Requires a separate discussion
  • VxLAN Control Plane, Data Plane & Policy Plane in Multi DC, Multi Cloud and Active-Active DC - Needs separate discussion
  • Control Plane & Data Plane Optimization in VxLAN Context - Broadcast Suppression, Conversational MAC address Learning,  BUM traffic (Watch out for those details in given implementation

So question remains - What value really VxLAN brings in the context of modern DC. (Please don't tell me scale, let's keep it as a topic of debate for another day) ?

Further Readings:




















Saturday, January 11, 2020

Why You Shouldn't Bother About Having An Inbuilt LTE Interface In Your SD-WAN Device



Very often I have seen Clients and Network Engineers being very much concerned about having inbuilt LTE/4G interface into their SD-WAN device. Here is my take on why it shouldn't matter or perhaps should be one of those things you should avoid. Because:

  1. They can hook external LTE Modem to SD-WAN device using ethernet hand-off. More importantly external LTE device can offer Dual SIM in Active Standby design from two different provider and even Active-Active setup.
  2. External LTE connectivity provider can offer their service on MRC (Monthly Recurring Charges) model.
  3. Most enterprises usually have partner/promotion plans from Mobile operators those can be leveraged.
  4. Makes deployment easy and consistent in Global Scenario. You don't need to chase several Telcos and keep track of usage etc.
  5. They offer few basic SD-WAN capabilities from their box natively.
  6. They offer external Antenna connectivity as well. Most DC floors running SD-WAN device with inbuilt LTE interface are likely to face Signal coverage issues sitting behind thick walls.
  7. Moving to 5G seamlessly becomes easy as you don't own HW and nor dependency on SD-WAN device itself
  8. Management interface/EMS to monitor, manage and troubleshoot Mobile connectivity centrally and central help desk model
  9. With inbuilt LTE interface you always need to verify Band and other compatibilities for given HW and SW version. External LTE interface through ethernet handoff takes that pain away and lower that risk.

Further Readings:


HTH...
A Network Artist

Thursday, November 7, 2019

The White Box Obsession in Enterprise Networking - An Attempt to Put End to The Madness

Image result for Whitebox

Network Disaggregation & White Box switching are one of those Talk of the Town things for quite some time. There are lot of startups in this space or niche players is one of those over hyped marketing terms to address that segment of OEMs besides other ones such as SDN, IBN & Cloud.

At surface they seem to bring couple of new factors into the equation which at first sounds very innovative such as:

Your Networking Hardware should be treated as Commodity similar to Server market. So idea is simple, you should be able to buy any X-86 based Networking gear and should be able to run NOS (Network Operating System) on top of your choice. In theory the NOS could be purpose built to meet your specific needs. So again , in theory you should end up paying for only the features of Entire NOS that you are using. Which at surface seems bit different from the way you used to pay to Networking Vendors in past. 

So essentially in theory you end up with:

1. Cheaper Hardware
2. Standardized HW which run on x-86 Architecture instead of on Proprietary Vendor Chipset
3. NOS (Open Source or Vendor NOS) that you can run on Whitebox
4. In theory NOS should also be cheaper as you might only pay for what you use

So on surface those promises are very interesting indeed. 

But...

If that's the case, why Whitebox idea is not that successful so far at least in Enterprise Segment ? 

Well let's look deeper and travel back into time to find some details. Where probably it all started ?

Well it started as different movements at different places. Some old school of thoughts around SDN are good starting points, it's just one of interpretation what SDN means for many still.

At the same time Webscales of the world were having different problems to address:

1. Pace of innovation - Now though Innovation interestingly is one of those over hyped and overly abused terms just like Digital Transformation. Let's say in simple terms in our context it means at what pace I can build features and function or integrations. 

Let's say your specific environment needs some very specific Add ONs on top of an existing feature (Though in theory you might want a new feature). Now this was and probably still is one of the most important consideration for large webscales and cloud providers as requesting these add on or features to vendors takes lot of time usually. The vendor on the flip side might not be at fault if you are big enough and giving them enough business or the frequency of these kinds of requests is not high enough. The moment you don't find that right balance, your requests will likely be pushed back or thrown onto potential road-map presentation slide.

The another roadblock could be compliance and privacy on both sides seen as an issue under co-innovation. 

But do you want to go that far as an average Enterprise ?

2. Who wants to keep the Control - Now this can also be looked at through different lenses and innovation can be seen as just a subset of it. 

Now let's look at simple perspective here:

Q: Who Pays your bills ?
A: Business/Job

Q: How Business makes money ?
A: By running Apps which are managed through People & Process which helps it make some money

Now you must understand that how fundamentally Webscales are different from Enterprises. At very basic level Webscales build applications first and later design and build infra to best suit those Apps. Also most APPs are written by Webscale staff itself.

On the flip side:
Most Enterprises have lot of Organic Growth. Where in you Build Infra first (The minimum capability) and run applications on top which in most cases are sourced from Vendors. So :

A. We pick infra first and later try to best lay applications on top.
B. You as an Enterprise don't have much access to make changes to Apps itself for most part. Depending upon your size you can either go through same cycle which I described under Point 1 or You can create and build abstractions on top which brings another set of complexities and considerations into the mix. 

3. Platform Independence - Now while the idea sounds interesting and marketing really seems to have done its job well on different social media platforms. Think deeply if it makes any difference in your context or if Networking Gear vs. Server Gear really needs to be compared. Try to map it back to your Organisation's IT strategy and see if it makes any sense. Another interesting problem here could be :

- If something breaks and you don't know if it's HW or SW, How you stop finger pointing which is one of Operational Challenge you may face. Now the possible argument could be Whitebox vendor eco-system as an answer to this problem. But wait, were we not looking complete independence here though Disaggregation ? So that's another tradeoff if you look closer.

4. Cheaper Hardware & NOS - Well x-86 boxes being cheaper looks promising. But you must understand most of your current vendors don't make money from selling Hardware anyways for most part. Depending upon your size as Customer and loyalty you bring on the table for your respective OEM, I have seen discounts anywhere between 60% - 80% and in exceptional cases 90% to 100% depending upon the play. A good example would be Vendor being into Software lead Networking solutions space. The vendor can also give you lots of free stuff initially under it's pull through strategy in other different ways for which otherwise you may end up paying heavily such as Professional Services. Also most modern & Traditional Networking Vendors offers their NOS under different subscription models and different licensing tiers. Which sets the balance here.

Now assuming you still want to continue with " Whitebox Strategy ", here is another good set of considerations:

1. RMA Strategy - Dig deeper into Whitebox vendor RMA strategy and SLAs as a relatively new player might not be able to match expectations. It's usually a larger problems for Organizations with Global footprint.

2. Quality of TAC Support - Assuming vendor is relatively new in the Industry, A good assumption would be their HW or NOS being not mature enough. Over period of time as they grow, people put their stuff into interesting environments and pushes the limits. Also any given OEM TAC would rely upon not only experienced staff but also how large your Database is. 

3. Existence - The rapid pace at which industry has evolved in last couple of years, we see some traditional players going out of business, being taken over by large players where niche players are bought by Vendors with deep pockets and M&A scenarios. So your favorite Whitebox vendor or NOS vendor may not last in Industry for different reasons. So ROI and Future protection are key considerations here.

4. Re-Skilling - Your existing Operational staff has likely no prior experience to operate this new way of thinking or working per say. But that's not where it stops, During the transition/migration you may end up maintaining this Hybrid environment (OLD + NEW) and if you haven't put enough thought process into this and got your SOWs, Run Books etc. revised it may cause more damage than good. And of course understand learning curve and it's tradeoffs. 

5. Platform Performance & Integration - Most enterprises usually at least in my experience don't have Platform testing teams in house and kind of rely on vendors which works based TRUST MODEL. From a traditional vendor perspective you at least have some level of control if at later stage the Gear doesn't meet the expectations. In a disaggregated world it kinds of gets tricky as HW and SW comes from two different vendors and Finger pointing likely doesn't go away. The moment you again apply the Eco-system logic, you run into another Anti-Pattern which was complete independence. 

6. Reach - Most people don't consider this but reach / ecosystem of System Integrators is another good consideration. Major SIs in market of course work with Major OEMs. Which means a relatively new SI may neither match the reach specially in Global context nor the quality of staff.

7. Experience - Being a new way of doing stuff, how would you compensate for experience

8. New Security Surface - Every platform and NOS has its own security surface which needs to be addressed by Organisation's (Client) IT Sec policy or Cyber Security policy whichever terminology you may prefer. How your favorite Whitebox and NOS vendors will fit into this existing ecosystem is an important consideration and most CISOs likely don't have first hand experience with such environments or not enough experience perhaps. So you as a Network Team is giving him/her some interesting work, but how many people like to work hard ?

So we can summarize that White Box is nothing new from Innovation or Engineering standpoint as such. At best it's a new fancy "Consumption Model" similar to " Cloud "

But in the end you must understand and articulate the real value in terms of Business Objectives you are going to meet with White Box and independent NOS strategy standpoint. If you don't do it correctly, this entire exercise will go for toss anyways.

Good Luck...

Further Readings:

https://packetpushers.net/research-towards-open-disaggregated-network-operating-system-att/

HTH...
Evil CCIE