Technology, Projects & a Yes Man Approach - The match made in hell (A Brief Writeup)

 

In tech you will often encounter people who have hard time saying "No" to their customers & even colleagues. Because people don't want to be seen as:

- Show stopper

- Negative/pessimistic

- Work dodger (in hindi we call them "kaamchor")

- Incompetent 

- Noise maker

- Uncareful 

In which case people often end up committing to asks from others which are - unrealistic, unjustifiable, impossible, impractical & last but not the least - out of scope.

So those people often:

- Fail at delivering their promises and commitments

- Make up things

- Channel incorrect and wrong information

- Fail at meeting quality and standards 

- End up overworking and experience burnouts

- Unable to find the right balance between work and personal life/family/friends/social & society bound activities 

But how do we handle it ?

Well it boils down to the Organization leadership and hence the culture. At least in my personal experience I haven't seen many people being successful to turn around the culture in corporate from bottom up. Have You ?

On the flip side You as an individual can still get better at it and play it right beside being ethical. So spend time (which most tech folks in general don't give a damn about surprisingly):

- Set high standards for yourself (If you don't care about it, who else would?)

- Be more strategic in your approach (Hardest thing for people in general)

- Pick your battles wisely & carefully (You have limited energy & time)

- Improve your negotiation skills (People rarely think about)

- Learn the art of saying "No" politely (You can learn that)

- Pick up on "Effective Argument" skills (Don't just argue, do it the right way)

- Spend time on improving communication skills (You can do all of above and still fail if you can't communicate it properly and convincingly - It not only about what you say but also how you say it)

- Improve your vocabulary

- And yet be prepared to be criticized until you find your right spot and right place (Like I said, its a culture and leadership issue with little to no control you have)

Further Readings

-  Why Are You Always so Negative?

-  Planning fallacy

-  Cobb's Paradox

-  How Unethical Behavior Spreads

-  You Can't Incentivize Performance  

-  Negotiation Skills

-  Effective Argument Skills

-  Effective Communication Skills

-  Effective Problem Solving Skills 

-  Handling Criticism 

-  Strategic Skills 

-  Improving business writing skills 

-  RFC 1925 Rule 3 & 6

-  Being Ethical Is Long-Term Greedy

-  "Clouds, Overlays and SDN: What really matters" Ivan Pepelnjak

-  The Three Paths of Enterprise IT

-  Why Is Public Cloud Networking So Different?

HTH...

A Tech Artist 🎨

An Architectural Perspective on Hierarchy In IP Networks - A Complex Puzzle Comprising Protocols, Topologies, Addressing & Systems (A Short Post)

 

If you ever bump into a Network Design book, course, blog or a webinar - most likely you are going to get introduced to this interesting & an important concept of " Network Hierarchy aka Hierarchical Networks Design Principle ".

Now depending upon which study materials and authors you follow, you would likely to come across different view points in terms of it's needs, pros & cons. Which in general not only contributes into more confusion among audience but also when I speak around with experiences Network Architects & Design Engineers, I often find them:

1. Having different interpretations of this concept and different view points

2. Considering this to be a very theoretical concept which you are likely to encounter in most Architecture & Design books but don't know about:

A. How to practice it (By applying theory to practice) 

B. How to measure it 

C. Missing the deep understanding of the topic at hand beside failing to understand its tradeoffs

So Idea behind this post is to offer you some architectural decision pointers to think through the problem statement and break it down into few tangible pieces by following another important network design principle " Separate the Complexity from the Complexity - Russ White " beside examining the rule 8 from RFC-1925

One of the model I personally always find handy is the SOS model from Russ White and you can use it too as a good ref. point.

So here is the quick list for you to think through in a more pragmatic manner:

1. What problems are you really trying to solve by introducing hierarchy into the Network (Go beyond theory) ?

2. Is it always possible to follow hierarchy? Specially in brown fields or during transitions (Think of old gear with still some lifetime left, mergers )

3. What are the downsides of introducing hierarchy ? (What harm it can cause and tradeoffs such us downgrade Agility, Flexibility, Organic Growth etc.)

4. Difference between Hierarchy vs. Symmetry vs. Modularity vs. Abstractions 

5. Different types of hierarchy/ layered approach to it ( physical level hierarchy,  logical level hierarchy,  hierarchy in addressing scheme, Protocol Level Hierarchy (ISIS Levels & Addressing ?) and so forth)

6. What data points you have in place to test your hypothesis to measure its impact on network

7. How these concepts are applied to different network environments - Enterprises (Campus <Wired and Wireless>, WAN/SDWAN, DC) vs Teclos vs CDNs vs Cloud Providers vs Web Scales vs Within public cloud virtual DC + Controller vs. Controller Less Architectures

8. Impact of introducing hierarchy on Visibility, Reporting and Performance mgmt. of the network

9. Impact on hierarchy on information hiding <reachability information> vs. topological information hiding (Aggregation vs. Summarization)

10. How all these choices will flow into your equipment sizing and potentially have an impact on your decision process

11. How will you apply all these concepts in a IPv10 network (IPv4 + IPv6 aka Dual Stack)

12. And if you are still brave enough :) , read through the further readings list to get to the bottom of this rat hole

Further Readings:

P-FatTree: A Multi-channel Datacenter Network Topology

Enabling Wide-spread Communications on Optical Fabric with MegaSwitch

Abstraction in Networks with Russ White

Hierarchical Network Design Overview

Engineer Versus Complexity

Optimal Routing Design

Navigating Network Complexity

Network Topologies

Five Number Summary for Network Topologies

Scaling MPLS Networks

The Side Effects Of Route Summarization

Avoid Summarization in Leaf-and-Spine Fabrics

Valley-Free Routing

Intra-Spine Links in Leaf-and-Spine Fabrics

Nonblocking versus Noncontending

Hierarchical IP Address Design and Summarization

Hierarchical IPv4 Framework

Fabric versus Network: What’s the Difference?

Liskov Substitution and Modularity in Network Design

Dragonfly+: Low Cost Topology for Scaling Datacenters

Reliability Basics- Part1

Network Centrality and Robustness

Swimlanes, Read-Write Transactions and Session State

Fifty Shades of High Availability

HTH...

A Network Artist 🎨

Overlay Networks & Protocols Tradeoffs - Aka SDN aka IBN aka Magic aka Silver Bullet

 

A long time ago I wrote a short article on what really went wrong with SDN, now a few years later the topic still pops up in a conversation with the great Ivan & he acknowledged my list of Tradeoffs (things to watch out for carefully) related to overlay networks and protocols which seems to be de-facto standard for most modern Network solutions that we see around in Enterprises & Telcos.

• Impact of overlay networks on visibility, reporting and performance management
• Additional control plane that would result in additional abstraction layers and interaction surfaces and hence cascading effect in many situations
• Impact on troubleshooting: how many solutions do we see in the market that can correlate underlay and overlay problems?
• When it comes to sizing equipment in terms of control plane or data plane, it poses a new level of complexity an architect would need to deal with and in most cases vendors themselves won’t be able to offer much help in general rather than just asking you to believe in their words
• I see lot of VXLAN and EVPN preachers, but let’s agree that mapping VLAN to VXLAN on 1:1 basis tells me you don’t know your stuff and believe too much in vendor marketing
• EBGP underlay with IBGP overlay…man we can do better
• Stitching two EVPN DCs with MPLS and SR: most of the implementations that I have seen were too complex and too fragile and thus results in a complex “policy.”

Ivan's Original Post

Further Readings:

Disjoint Path Routing and LP

HTH...

A Network Artist 🎨

Marrying SASE (SDWAN) with 5G - The Marketing, The Myths & The Fallacies & How to Get it Right (An Architectural Perspective)

When almost 3 years ago I wrote about why having an inbuilt LTE interface inside a SD-WAN device doesn't really matter, the 5G thingy was still relatively new.

During a recent Enterprise Architecture Consulting engagement, I was asked by one of my client if they should really care about 5G and 5G interfaces on the variety of SD-WAN platforms that were pitched to them by different Systems Integrators (SIs) & MSP (Managed Service Providers)/Telcos.

So let's start with a simple question - "What problem we are trying to solve?"

In general you will see a few types of customers in SASE/SD-WAN market :

1. Which have Technical/Solutions architects those are completely sold on vendors marketing (50% of the crowd)

2. Those who wants to jump on the bandwagon due to fear of being left behind in the similar industry or by the competition (25% of the crowd)

3. Those who always are either too excited by technology or have a lot of money to throw onto the problem (the next 20%)

4. Those who can really map business capabilities to technology capabilities (the rare and the last 5%)

In general, you would often find a few ways the 5G gets included into the solution by solution providers such as :

Design 1 - You have a site (mid/large size) which either has got hybrid connectivity (MPLS + Internet) or 2 x Internet connects, while keeping 5G cellular as a last resort backup link in an event of a total failure.

Design 2 - A small site that usually runs on a single internet link and keeping 5G as back for last resort.

Design 3 - 5G as backup of last resort onto your DC WAN edge

Now in general it doesn't look like a bad idea to have a 5G interface but here are few of the important considerations:

- In general your DCs/COLOs and Campus networks would be setting on racks behind thick physical building structures (remember your Wi-Fi coverage problems even while your APs are sitting inside), so very often you would expect coverage issues. Now one of the argument here might be that I can install an external antenna of some sort on top of building or floor and run a fiber cable connection from there. Fair...but:

> Now you got to get approval for installations, cable runs, have administrative processes and safety processes in place and what not (What if the lightening hit the antenna?).

- To my understanding, most SASE/SD-WAN solutions don't offer any visual monitoring, reporting & troubleshooting tools for - Checking 5G signal strength, 5G interface troubleshooting, Dummy traffic probes etc.

- Interestingly enough now you need even a more complex traffic distribution, traffic prioritization, traffic failover, traffic desired SLA/performance metrics and other set of policies into the mix. And even if you end up doing that successfully, how you are going to document it for the operations ?, Is your EMS/NMS equipped with such capabilities ? 

- From the network architecture perspective, you just added an another layer of complexity. Assuming this 5G interface is an HW module, you got to now deal with: New stack of software and protocols within your fancy WAN edge device (3GPP standards), New interaction surfaces, New potential grey failures.

- You just end up adding the more state into the network (State, Surface & Optimization tradeoffs) 

- You need to now have life cycle mgmt. in place for your 5G interface (HW/SW Upgrades, Monitoring, Management, Refresh etc.) beside that fact that the 5G specifications often vary country to country (even from Telco to Telco) and now you need to keep track of Data Plans, Data Usage, Availability & Performance Mgmt., Cost Mgmt. and what not. (Remember your data plans are pretty limited in general?)...imagine to solve these problems at a global scale deployment dealing with different MSP.

> How to you move to 6G if it comes out in the next few years ?

Now after all these interesting questions, we may still ask:

1. What are the better alternatives today ?

2. Where 5G might still make sense ?

Answering the first question, IMHO I would still recommend you to opt for a broadband connection and avoid 5G unless you have a very particular problem or scenario because:

- With broadband you are still dealing with Ethernet connection between your CPE and broadband router/device which is a pretty familiar connectivity model and protocol stack to deal with.

- In general your broadband data plans are much bigger 

- You don't need to deal with another MSP for service management perspective as in general your ISP would have a common portal to give you a view of all MPLS, Enterprise Class Internet and Broadband based Internet connections. (Also mind that historically your SPs were building two parallel networks for ISP (Internet Service Provider) and MSP (Mobile Service Provider) business units, though those are converging now more and more)

- With everyone hit by pandemic that accelerated WFH culture, in both developing and developed countries you would expect to have broadband being available very easily and at the affordable prices. The only places you might still face availability issues are Tier-3 cities and so forth. But for most part that is not a technology problem but your SPs wanting you to stick with cellular connectivity to drive more profits (So its an intent problem rather)

- My own research suggests that in many developed countries getting a broadband will cost you far less compare to the enterprise class 4G/LTE/5G/Private 5G connection

Answering the second question, here are few use cases for your consideration:

- 5G as part of your SD-WAN transition plan (Legacy to SD-WAN)

- Your last mile hybrid or other wired connectivity model still runs on same fiber or shared media (true path diversity problem)

- One or both of you last miles are on Microwave (Should be pretty rare now)

- 5G as a OOB (Out of Band) management option

- Movable workplace and offices (Eg. Marketing/Sales & Promotion offices or moving semi-trailer trucks often used in variety of businesses)

- Specific operating conditions such as in Oil/Gas & Mining industries

- Edge compute/Cloud

- IoT Platforms

HTH...

A Network Artist 🎨

Further Readings:

SD-WAN Leads to $96,000 4G/LTE Bill

Improve Your Home Internet Performance Using CoDel

More Bandwidth Doesn’t Matter (much)

Are Networks Really Complex ?

Enterprise QOS Design & Deployment - Good, Bad Or Ugly ?

Focus on Your Business, Not Fancy Technologies

Are You Solving the Right Problem?

This Is What Makes Networking So Complex

Are Business Needs Just Excuses for Vendor Shenanigans?

The Three Paths of Enterprise IT

SDN Will Not Solve Real-Life Enterprise Problems

Why Intent Based Networking (IBN) will Not Save Your Network Anytime Soon ?

Complexity and the Thin Waist

It’s Most Complicated than You Think

Details and Complexity

A Simple Routing Protocols Decomposition Model - Part 2 (Peering Mgmt.)

 

So in the first part of the series, we started with rather a simple decomposition model to get bit more insights into the routing protocol internals.

So let's continue the series by expanding on the very first layer in our model - Peering Management.

In any routing protocol before we get fancy in terms of which all features, functions and knobs to use, the very basic requirement is to peer with other devices into the network since eventually a routing protocol is nothing but a distributed database. Routing protocols are designed to convey different set of information to its peer devices such as:

- Topology Information

- Reachability Information

- Policy Information

The type of information that a given routing protocol would exchange with its peers would largely depend on the protocol itself (OSPF, IS-IS, BGP) & where that proposed routing protocol is used into the network (Campus, WAN, Metro-E, DC) as implementation specifics do change.

[ Click on Image to Enlarge ]

As you may notice, there are lot of things working behind the scenes when it comes to peering in a routing protocol context. But don't get carried away by looking at the complexity. Once you look closely, all of these pieces kind of makes sense.

So let's start with the top row, reading it from left to right. 

Self Identity - Before the routing protocol determines whom it needs to communicate with and what information needs to be exchanged, it must find its own identity first. The most common way to give an identity to the routing protocol instance itself is assigning it a router id (RID). The RID can be configured manually or it can be derived automatically depending upon the platform and NOS. 

In real life assuming network virtualization is much more common today, you are allowed to configure unique RID for each routing protocol as well as a unique RID for each instance/process under same protocol. 

Protocol Addressing - Once the routing protocol is able to define its identify with a RID, the next step is to understand it's addressing. The addressing serves many purposes but the most basic one is to provide location services from the overall network view standpoint. Though protocols such as LISP was an attempt to separate device identity from device location, due to limited use cases (such as mobility) and other problems it never really took off well.

Every routing protocol has it's own addressing scheme which may further have impact on its scaling and expected working behavior if not done correctly.

Participation - The next step for the routing protocol is to determine its participating interfaces on a given device and in certain cases the entire device itself. Depending upon the design you may run into some interesting challenges though.

Reliable Transport - Every routing protocol needs a reliable transport to be able to effectively communicate with its peer device. The reliability itself is an important aspect and while some routing protocols such as BGP rely upon existing TCP stack, others like OSPF uses IP protocol 89 & EIGRP choose its own transport protocol namely RTP.

Neighbor Discovery - In the next step the protocol must discover its peer/neighbor/adjacent device depending upon which routing protocol you are following. The protocol while ideally should keep a track of its neighbor and relationship state, this may or may not be implemented.

The neighbor could be configured manually or dynamically discovered, while the discovery phase itself may use unicast or multicast as a transport for reachability purpose to the next hop device. Also the reachability to the neighbor could be over layer 2 transport or layer 3 transport depending upon the protocol. IS-IS and many other IOT industrial protocols operate at layer 2 for example. In case of eBGP, the neighbor in fact may be multiple physical hops away.

Neighbor Identity - While we may have discovered our neighbor, it doesn't mean the neighbor itself is a an intended neighbor or a legitimate neighbor always. After all someone might want to spoof or sometimes we may end up discovering somebody completely un-intentionally. So sharing any information with an unexpected neighbor won't make any sense. To prevent this we have several measures which we can put in place such as Authentication, Validating neighbor's identity (remember they also have RID), Validating neighbor based on IP Packet's TTL value etc. such as in OSPF & BGP. The modern day solutions such as SD-WAN usually uses RPKI over TLS/DTLS channel.

Establish Session - Once the neighbor is discovered and validated, we finally establish a session with it. Depending upon the protocol, we might have a single session vs. multiple sessions going on. A simple example would be networks running IPv4 & IPv6 at the same time under single routing protocol instance. While some implementations exchange information related to both IPv4 and IPv6 over a single session, some may do it over a separate dedicated session for each of them. Long time ago there was an attempt to run multi session bgp for MTR (Multi Topology Routing) for network virtualization use cases.

Capabilities Exchange -  This is an another interesting step where the routing protocol running on separate devices exchange capabilities with each other to find the lowest common denominator. For example we know BGP is more like an application that runs on top of TCP as opposed to a pure layer 3 routing protocol which only carries routes. BGP though can carry layer 3 routing information and in case with most vendors it's been the default behavior, BGP does allow us to carry many other set of information depending upon the use case in the form of AFI/SAFI which is essentially an encoding format. For example BGP can carry MAC Addresses information under Layer 2 VPN EVPN address family when enabled. Though with BGP, you got to be cautious about enabling a new capability/address family in production network as highlighted here.

Establish Adjacency - The protocol reach this far and finally the peers are ready to exchange required set of information needed to populate RIB & other details such as topology graph. An interesting example in a routing protocol context would be (In case you are wondering in which scenario two devices would be neighbors but not adjacent) OSPF.

Messages Exchange - Finally we [assuming by now you are thinking like a routing protocol :) ] reach this stage where in we finally start exchanging information through messages. The messages needs to be sent reliably, keeping track of to understand which one to prefer in case of being received from multiple sources, acknowledged and so forth besides how to queue and dequeue them and at what intervals those should be sent out vs. being hold back for a while to pack multiple events together for optimization and getting the latest information being sent out.

Further Readings:

Cisco IP Routing: Packet Forwarding and Intra-domain Routing Protocols: Packet Forwarding and Intra-domain Routing Protocols 

Network Routing: Algorithms, Protocols, and Architectures

Network Algorithmics,: An Interdisciplinary Approach to Designing Fast Networked Devices

Inside Cisco Ios Software Architecture

HTH...

A Network Artist 🎨

A Simple Routing Protocols Decomposition Model - Part 1

 

You might have heard about this term "Mental Models" earlier too. Mental models are essentially a very simple yet very powerful tool. At its core the purpose of mental models is to provide you with necessary tools and building blocks in order to understand how something is really built and how it works behind the scenes.

Whether we recognize this or not, we all have mental models of some kind in our mind that we use and apply to situations in our daily life. It's just that some of those are developed consciously to solve certain problems or how to we approach certain things, on the other hand some of those are genetically coded into our DNA by nature.

A quick example of consciously developed mental model would be "How do you manage your finances" while the unconscious one would be to "Run fast when you see a lion charging towards you".

The mental models can be simple or complex, layered and even blend of multiple models depending upon how complex the topic is at hand and how deep you want to go down the rat hole beside in some cases tracing down the roots of the real problem into its adjacent domains.

Last but not least, your experiments and experiences always help you enrich you mental model aka "Lesson learned the hard way". On the flip side mental models do create "biases" if totally ignored, as one of the tradeoffs to follow them. But we will keep that topic besides the other tradeoffs for another time.

Now you might be wondering:

1. What is than Decomposition Model

2. Do we have some of those being available for IP Networking

To answer the 1st one - The famous consulting firms were looking for some more glamorous term which would resonate better with business people, after all technology details looks boring to them for most part and they started calling it "Decomposition Models" and heavily use this term in various cases such as for "Maturity Model" .

Coming back to 2nd one which is more relevant for this conversation, IP Networking actually taught us few mental models at the very beginning of our career as Network Engineers in the form of - OSI Model, TCP/IP Model, RINA Model, Hourglass Model and so forth. But as we advance further into our journey, the numbers of models available to us starts to shrink if not completely disappear with an ultimate answer to every question not as the "number 42" but the magical two words " It depends ". Standard bodies such as IETF, IEEE, MEF & ONUG etc. don't offer much help either for the most part.

But then nothing stops you from to being a little more creative and come up with your own models.

Let's start with a " Routing Protocol Decomposition Model ".

It's a multi-purpose simple yet effective model that you can consider for :

- As a Routing Protocol Designer it allows you to breakdown the complex equation into smaller chunks

- As a Network Architect it allows you to breakdown you design choices into smaller & distinguishable sections/containers and allow you to tune the variables to achieve certain outcomes while having a better view of "Tradeoffs" that you are going to make as part of the process

- As an Implementation Engineer it allows you to breakdown the device configuration into logical containers which are easier to manage, understand & configure beside without falling in the trap of - Order of operation issues

- As an Automation Engineer (NetDevOPS, NetOPS, NetSecDevOPS of whatever else you may call it) it allows you to break down the protocol in such a way that you can now easily write a Data Model to code it, write test units for each individual block and so forth beside develop a "YANG Model" or build a " CICD Pipeline "

- As an Operation Engineer or TAC Engineer it allows you to approach the whole process more logically if you follow the direction of "Arrow" as it allows you to understand "Dependencies" as part of layered approach

- As a Systems Architect it allows you think clearly about " Interaction Surfaces" & " Leaky Abstractions " beside understanding " Dependency Relation "

- Helps you form an " Abstraction Model " to hide complexity & inner-workings/details

- It allows you to plan " Streaming Telemetry " better as you can map your use cases and dependencies more easily and feed it into a magical " Intent Based Networking (IBN) " solution

- A consultant can use this for "Maturity Modelling" & " Assessment/Observability "

- A tool to apply " Critical Thinking " to routing protocols

In next part of this series we we bring this model from current 10,000 feet view to 1,000 feet view by start populating sub blocks under each major block.

Meanwhile as an exercise, try to think of any feature or knob of your favorite routing protocol as see if there is anything you can think of that wouldn't fit into any of those layers. :)

HTH...

A Network Artist 🎨

State of Networking Industry In 2021... a bit of sarcasm with a pinch of salt :)

 
- Network Engineers those are still using "CLI" are "CLI junkies" & "dated"

- Network Engineers those switched to GUI in the holy name of SDN/IBN Solutions are " cool "

- Network Engineers those are often writing scripts in YAML & JINJA are "coolest"....shall we call it "The Holy CLI Mode" ?

- Network Engineers using "Machine Learning" are " Gods of the thunder"

And now You can safely forget about - Robust Network Design & Implementation, Standardization, Modularity & Failure domains, Statistical Analysis, People-Processes-Tools

HTH...

A Network Artist 🎨

The Reality is Not As You See It - Network Engineering Pro Tips

In Enterprise Networking context, 90% probability is that You are never going to encounter " Multi Area OSPF ", " OSPF Virtual Links ", " OSPF as PE-CE Routing Protocol ", " EIGRP as PE-CE Routing Protocol " & " BGP Confederations ", assuming " RIP " is by now resting in peace ✌

Unless...

1. You want to be fancy
2. You are preparing for your favorite vendor's expert level lab exam
3. Your network design is broken
4. Perhaps you have really good and convincing reasons to opt. for odds

And of course its worth knowing - how anti-patterns such as SD-WAN and MPLS Layer-3 VPNs makes it even rare besides tons of overlays you see around.

So next time you see those on your favorite Networking vendor exam blueprint, don't shy away to ask ...Why ?

HTH...

A Network Artist 🎨

" How to better prepare for interviews "

 

- Every interviewer and every company has patterns & biases, the sooner you find it out, better off you would be.

- There are only two types of interviews, in first case you take everything in control as much as possible from the beginning to the end. To get to that level - invest more time learning about: Mental Models, Human Psychology & Biases, Story Telling, Presentation Skills & Marketing. In second case - the person on other side of the table takes control and take you for a ride.

- Preparing for interviews is a serious business for people serious about their career or at least cracking the interviews.

- While most interviewee care and think more about what they can get from given company and role, flip the question in terms of what you can offer and will bring new on the table - with the later mindset you will perform much better.

- At the end of an interview, make sure you ask questions to better understand your role, work culture, particular expectations that the hiring manager may have, what systems and methods they have in place to measure and rate employee performance but more importantly business performance in terms of business line you will be part of.

HTH...

A Network Artist 🎨