Why would I care " How does that God Damn Router Really Work ? "

 

    

Interesting enough, You could pass your favorite networking industry vendor's Expert level lab exam and through out the process, they never really tell You how does Router/Switch (or whatever fancy name they may have) really work behind the scenes.

On the flip side, at its core - IP Networking is all about moving packets:
1. From point A to point B
2. Switching packets between Input & Output interfaces

But still You can provide an expert opinion on Control Plane, Data Plane & Mgmt. Plane

Perhaps You believe too much into voodoo magic  😉

HTH...

A Network Artist 🎨

The Tale of Application Aware Networking

 

Almost a decade ago some Networking Industry vendor/people threw this idea of the holy " Application Aware Networking/Infrastructure " thingy and still today it seems of a popular school of thoughts among Network Engineers that follow the Flat Earth Theory.

What they often miss though is:

1. Business runs on top of business platforms, business processes and services as opposed to contrary belief on Applications.

2. Applications comes in different forms, shapes and Architectural choices - SOA, Microservices, Centralized, Distributed etc.

3. If you keep track of Applications or meta data, You must keep and maintain their state which means resources and state sync

4. You as networking engineer have never been taught about how to gather Application requirements

5. Your favorite OEM/Vendor doesn't provider you with any of good tools, procedures and methods to capture any of above information effectively.

6. Do you speak the language that Business folks understand ?

HTH...

A Network Artist 🎨

Da Vinci Code of IT & Network Engineering (Part 1 of 5) - A Road to Clarity ?

Now the reasons for writing this post were many. But let's list them down quickly and briefly before we move forward, since in the end of the day people who are reading this should have good clarity if this is of their interest or not without diving too deep. 

1. Lately I'll be completing 18 years into the current IT industry this month. While It may sound like a lot, but eventually I am still learning pretty much everyday. Some of those learnings comes from - Books, Blogs, Videos, Audios, Research Papers, Practice, Discussions etc., while others came and still are from - First hand experience, A continuous closed loop of -> Learning -> Practicing & Experimenting -> Build Mental Model -> Validating Hypothesis -> Improve & Optimize. 

Besides this I definitely met some great people. While many of those from same industry and profession that I admired (More on this in part 2 of series), in last couple of years I rather met many other people which were from different industry and gave my different and broader perspectives around things that I never thought much about myself early on in my career.

"The people who are very much like You (Same industry or profession), will help you become bigger version of yourself. But people who very different from You (different industry or profession), will help you become better version of yourself".

2. Lessons learned "The Hard Way"

3. There is a lot of non-sense that is going around in the IT and Networking Industry at the moment and lately tons of people keep reaching out to me to find about how they should sail throw this ocean of dilemma around - what to learn and what not to learn, since there are just too many things out there. So Yes, hopefully by the end of series you will come out of it clean & clear.  

4. Lately many people, colleagues & friends in IT and Network industry lost their job in this pandemic (#OpenToWork is pretty common on LinkedIn). So on my part I started an initiative to help those people as #OpenToHelp , So I am hoping they find this series useful.

5. It's my way of paying back to IT & Networking community.

So buckle up and hold on tight...

HTH...

A Network Artist

Are Networks Really Complex ?

A recent short article from an IBN Evangelist brought up an interesting point about how complex modern networks really are and how IBN (Intent Based Networking), assuming SDN is dead, would potentially transform your network and equip you with enough insight by virtue of using AI & ML in such a way that overall network will be much more predictive in nature & AI/ML magic will take care of all complexity.

Wow...by now if You are not sold to this such a great marketing sales pitch....I'll be disappointed 😈

But as an Individual Network Consultant & as a Network Architecture practitioner, I often help my Clients to look deeper under the hood and uncover insights which helps them transform their networks in such a way that's much more realistic and much more practical in nature.

So let's start uncovering some details around " What makes our Networks really Complex? " to understand the problem space at its core and not just at its surface and bring this idea to " How to go about this idea and bring it to reality".

Business Layer

+++++++++++

Often business people based on tons of marketing from vendor believe that the technology is and was the way to transform their businesses. And don't get me wrong but that's true for most part at it's surface. But on the flip side what they don't get it is that the Technology is part of solution and not heart of solution. At it's core you still got to get it right from People & Process perspective. That's one of the reason why over 80% of Digital Transformation projects which are technology centric often gets failed and they get failed miserably from ROI perspective.

To simplify this I often quote this to my friends and colleagues " In theory your Smart phone has more computing power than Apollo 11 which took men to moon, but doesn't that mean you know completely how to harness this power still today to it's potential"

So why most technology driven Digital Transformation Projects really fail today ?

1. Short Term Thinking 

2. The Technology Fallacy - At its core in any business its the People make the difference not tech

3. Absence of Collaboration Culture - Start building trust at its core

4. Imposed Fake " Innovation " Culture - It's a big one in big Orgs.

5. Vendor driven Tech & Innovation - Vendor religions are still ruling the world

6. Most Valuable yet underrated skills:

    A. Common Sense

    B. Listening

    C. Hard Work

    D. Framing the Problem correctly and map it to its roots 

    E. Hypothesis Testing of your ideas, thoughts and frameworks 

7. Fake Data Analytics - People using biased data & statistical models to prove their point and sound intelligent

8. The Generation Gap - This is actually a very big one

9. Absence of a comprehensive & mature "OEM/Partner/Solution benchmarking & onboarding process"

10. Moving away from Vendor A to Vendor B to save cost - If it's just CAPEX, just don't do that

11. OPs & Engineering are usually not invited in Business reviews as they seem irrelevant to many business folks still today

12. Most cross skill programs are fake and misleading

13. Too much of belief in - Cloud, SDN, IBN, Automation and the saga continues...

14. Believing that it's eventually the APP that matters and not the underlying tech...remember it's still Information Technology (IT) and not App Technology (AppT)

15. The urgent culture - Remember that most urgent things are unimportant 

16. Let's not focus too much on those MBAs and CAs running companies at this point 😇

Technology

++++++++

1. The fake Application Aware Networking notion - It doesn't exist...Period ! 

2. Most Networking Managers & CTOs are still not investing into Core Network Architecture & Design skills and keep believing in OEM's Voodoo magic

3. Lack of Test bed environment -  Over 90% of organizations still don't own dedicated Test bed environment that can validate those fresh configuration & configuration changes by replicating your production network at scale

4. The fake Architectural Skills - Just because someone owns an Expert Level networking certification or tons of experience on resume doesn't mean they own strong architectural & design skills

5. The old school & immature Interview Process

6. Common misunderstanding that if someone is working on new cool shiny tech, they are innovative

7. Tech people still believing understanding "How business really works" is boring & unimportant since that's why they chose to become Engineer 

8. Vendor & Technology/Solution religions 

9. All you need to pick on a new vendor and it's solution is that 3-5 days of training - Really ?

10. Tech folks still thinking in SLAs which BTW is a business metric and not a technology metric to begin with while still keep ignoring SLOs and SLIs

11. Tech Folks not being able to articulate changes they would need around People & Processes when introducing new shiny Tech, So how you handle change management in your SDN solution world and most importantly how different it is compare the what it was before introducing SDN ?

12. Outdated and non-standardized network documentation 

13. The One Book Mentality - Not sure how many times and how often I came across his mentality. Usually the moment you ask a regular network engineer for example "How should we go about implementing IPv6 in our network ? ", the common set of responses you will hear back would be:

 A. Let me read a good book on this and come back

 B. Tell me what you want me to implement

 C. Let me find the IPv6 deployment best practices document

But often they miss the point of first get a full understanding of following to fully understand the problem statement:

 A. Why we are doing it in first place ?

 B. What problem we are trying to solve ?

 C. What changes we would need to make across Tech, People & Processes ?

 D. What are my alternatives to meet same goal ?

 E. How we would measure our success and more importantly our Business will measure it for us ?

14. Most Networking tech folks are still madly in love with Vendor stories & narratives when it comes to learning

15. Learning a product doesn't make You an Engineer, understanding how tech works behind the scene does

16. No tight & close feedback loop between Design & OPs team

Standard Bodies Failures 
++++++++++++++++++

Like every other Industry, we got IEEE, IETF, MEF & ONUG and bunch of other standard governance bodies in Networking. But how robust their process are at their core ?

1. In most standardization documents, its not mandatory for You as author to talk through entire life cycle bits of your proposed protocol or solution. Always remember that building something new is perhaps always the easier part compare to operating it for next 5 years or so on daily basis. Under the hood the problem is very similar to " Why most hackathons are useless " for most part despite being chosen as one of popular tools to drive innovation. 

2. Most RFCs are just boring lengthy texts...Not saying all of them since I am in love with RFC 1925 and RFCs do add lot more details in the equation, but why can't those be written in more interactive manner in a modern world where people predict that visual and AR/VR based learning is going to be the future despite we ignore based on evidences that people retain visual memories for longer. Perhaps they put on a dedicated track around this lead by Dan Roam at some point understanding its importance.

3. Networking & even Expert Level certifications only focus on Tech aspect and are still far from reality you have to face in real world...ping me to discuss those numerous examples 😇

4. Security aspects of  any new protocol or solution is still an after thought in most cases

5. Lack of metrics defined to measure the quality of any protocol, solution & tech

Let's keep Complexity Theory & " Complex vs. Complicated " for some other time 👽

So what do you think now about " Are Networks really Complex ? "

Further Readings:

The Hard Thing About Hard Things

Navigating Network Complexity: Next-generation Routing with SDN, Service Virtualization, and Service Chaining

Your Strategy Needs a Strategy: How to Choose and Execute the Right Approach

Cloud Repatriation: Why Are Workloads Coming Back Out Of The Cloud?

High-Level Technology Guidelines

HTH...

A Network Artist

BGP Nuts - The Crazy Metric Story

Recently ran into an interesting BGP behavior. In order to convey the issue, I have oversimplified the design.

R2 (AS 100) has a network host 22.22.22.22/32 which is trying to reach R7 (AS 200) attached network host 77.77.77.77/32. 

As you can see, between AS 100 & AS 200 we have multiple exit points for the purpose of redundancy. However the interesting twist here is that those BGP next hops are learned from different IGPs.

54.0.0.5 <- Learned from EIGRP

63.0.0.6 <- Learned from OSPF

Given the scenario, what you think about which BGP route and next hop would make into the routing table ?

Well...most likely the answer would be BGP next hop learned from EIGRP because (Assuming... 1. " no auto-summary " is now default for EIGRP (resulting into same Prefix Length for route) , 2. EIGRP AD is 90 compare to OSPF Intra Area route which has AD 110. So based on theory how Router Process this information, the EIGRP learned Next hop (54.0.0.5) should be the one picked up by BGP Best Path Selection Algorithm and should be picked up as the best route over OSPF learned Next Hop (63.0.0.6)

All good ? ... Let's look at the BGP RIB to find it out and How IGP next hops are processed based on standard BGP Next Hop Processing approach that we know from standard BGP theory. Also let's run a quick traceroute towards 77.77.77.77 sourcing from 22.22.22.22 on R2

Now sure if you expected that...right ? :)

From BGP RIB standpoint we can clearly see that BGP route that's making into routing table & FIB essentially is the one that has OSPF learned BGP next hop.

Here is what our OSPF database looks like

Let's do a basic failover test by shutting down interface g6/0 on R3 

So things works fine and as expected here, so let's "un-shut" the interface in order to dig deeper into the behavior ( a little unexpected one)

So why this behavior ?...Well this is where BGP plays a little dumb. It actually ends up comparing METRICs across two IGPs since it doesn't have view of EIGRP Topology Table and OSPF Link State Database. Now if you go back and check IGP metrics for next hops learned from EIGRP vs. OSPF, The metric for EIGRP route is pretty high (in numeric ) as 3072 vs. OSPF route as 2 (Refer to first CLI screenshot in the post)

Now what if we make the OSPF metric look bigger than EIGRP by manually increasing OSPF cost to see if that would allow R2 to install EIGRP learned route as best route

And Yes it works as expected this time too :)

Now for fun let's make the metrics look same by again changing the OSPF cost manually

Well that works too as expected. Now interesting question here is - Since IGP metrics learned from OSPF and EIGRP look same, can we turn ON " i-BGP Multipath " and achieve load balancing.

Let's try that 

Well that works well too and as expected.

One of the problem incase you haven't noticed yet with this design is - Assume the path through OSPF has many more hops added before you reach to the OSPF learned BGP ASN exit point. Since OSPF cost metrics are much lower value compare to EIGRP, You will end up taking sub-optimal path from R2's standpoint in this case which may not be desired behavior. Alternatively You can replace EIGRP with IS-IS and you will by default still end up following OSPF learned BGP next since in most IOS version 10 is the default IS-IS metric for each interface hop. Obviously in practice you should try to run Single IGP across AS 100 here to avoid such issues though M&A scearios are always interesting and challenging. :)

Try this with " DMZ Link-BW " for Unequal Cost Multi-Path and I am pretty sure it will be fun. BGP AIGP NLRI is another interesting bit if you care depending upon the design. 

And of course, the above mentioned logic doesn't apply to following design where we have single exit point being reachable through both IGPs

To learn some more around BGP anomalies and somewhat un-predictable behavior:

BGP Wedgies

BGP Anomalies 

And if You want to master BGP from Design Standpoint, I would highly recommend " BGP Zero to Hero Design Masterclass " from my friend Orhan Ergun

HTH...

A Network Artist