Marrying SASE (SDWAN) with 5G - The Marketing, The Myths & The Fallacies & How to Get it Right (An Architectural Perspective)

When almost 3 years ago I wrote about why having an inbuilt LTE interface inside a SD-WAN device doesn't really matter, the 5G thingy was still relatively new.

During a recent Enterprise Architecture Consulting engagement, I was asked by one of my client if they should really care about 5G and 5G interfaces on the variety of SD-WAN platforms that were pitched to them by different Systems Integrators (SIs) & MSP (Managed Service Providers)/Telcos.

So let's start with a simple question - "What problem we are trying to solve?"

In general you will see a few types of customers in SASE/SD-WAN market :

1. Which have Technical/Solutions architects those are completely sold on vendors marketing (50% of the crowd)

2. Those who wants to jump on the bandwagon due to fear of being left behind in the similar industry or by the competition (25% of the crowd)

3. Those who always are either too excited by technology or have a lot of money to throw onto the problem (the next 20%)

4. Those who can really map business capabilities to technology capabilities (the rare and the last 5%)

In general, you would often find a few ways the 5G gets included into the solution by solution providers such as :

Design 1 - You have a site (mid/large size) which either has got hybrid connectivity (MPLS + Internet) or 2 x Internet connects, while keeping 5G cellular as a last resort backup link in an event of a total failure.

Design 2 - A small site that usually runs on a single internet link and keeping 5G as back for last resort.

Design 3 - 5G as backup of last resort onto your DC WAN edge

Now in general it doesn't look like a bad idea to have a 5G interface but here are few of the important considerations:

- In general your DCs/COLOs and Campus networks would be setting on racks behind thick physical building structures (remember your Wi-Fi coverage problems even while your APs are sitting inside), so very often you would expect coverage issues. Now one of the argument here might be that I can install an external antenna of some sort on top of building or floor and run a fiber cable connection from there. Fair...but:

> Now you got to get approval for installations, cable runs, have administrative processes and safety processes in place and what not (What if the lightening hit the antenna?).

- To my understanding, most SASE/SD-WAN solutions don't offer any visual monitoring, reporting & troubleshooting tools for - Checking 5G signal strength, 5G interface troubleshooting, Dummy traffic probes etc.

- Interestingly enough now you need even a more complex traffic distribution, traffic prioritization, traffic failover, traffic desired SLA/performance metrics and other set of policies into the mix. And even if you end up doing that successfully, how you are going to document it for the operations ?, Is your EMS/NMS equipped with such capabilities ? 

- From the network architecture perspective, you just added an another layer of complexity. Assuming this 5G interface is an HW module, you got to now deal with: New stack of software and protocols within your fancy WAN edge device (3GPP standards), New interaction surfaces, New potential grey failures.

- You just end up adding the more state into the network (State, Surface & Optimization tradeoffs) 

- You need to now have life cycle mgmt. in place for your 5G interface (HW/SW Upgrades, Monitoring, Management, Refresh etc.) beside that fact that the 5G specifications often vary country to country (even from Telco to Telco) and now you need to keep track of Data Plans, Data Usage, Availability & Performance Mgmt., Cost Mgmt. and what not. (Remember your data plans are pretty limited in general?)...imagine to solve these problems at a global scale deployment dealing with different MSP.

> How to you move to 6G if it comes out in the next few years ?

Now after all these interesting questions, we may still ask:

1. What are the better alternatives today ?

2. Where 5G might still make sense ?

Answering the first question, IMHO I would still recommend you to opt for a broadband connection and avoid 5G unless you have a very particular problem or scenario because:

- With broadband you are still dealing with Ethernet connection between your CPE and broadband router/device which is a pretty familiar connectivity model and protocol stack to deal with.

- In general your broadband data plans are much bigger 

- You don't need to deal with another MSP for service management perspective as in general your ISP would have a common portal to give you a view of all MPLS, Enterprise Class Internet and Broadband based Internet connections. (Also mind that historically your SPs were building two parallel networks for ISP (Internet Service Provider) and MSP (Mobile Service Provider) business units, though those are converging now more and more)

- With everyone hit by pandemic that accelerated WFH culture, in both developing and developed countries you would expect to have broadband being available very easily and at the affordable prices. The only places you might still face availability issues are Tier-3 cities and so forth. But for most part that is not a technology problem but your SPs wanting you to stick with cellular connectivity to drive more profits (So its an intent problem rather)

- My own research suggests that in many developed countries getting a broadband will cost you far less compare to the enterprise class 4G/LTE/5G/Private 5G connection

Answering the second question, here are few use cases for your consideration:

- 5G as part of your SD-WAN transition plan (Legacy to SD-WAN)

- Your last mile hybrid or other wired connectivity model still runs on same fiber or shared media (true path diversity problem)

- One or both of you last miles are on Microwave (Should be pretty rare now)

- 5G as a OOB (Out of Band) management option

- Movable workplace and offices (Eg. Marketing/Sales & Promotion offices or moving semi-trailer trucks often used in variety of businesses)

- Specific operating conditions such as in Oil/Gas & Mining industries

- Edge compute/Cloud

- IoT Platforms

HTH...

A Network Artist 🎨

Further Readings:

SD-WAN Leads to $96,000 4G/LTE Bill

Improve Your Home Internet Performance Using CoDel

More Bandwidth Doesn’t Matter (much)

Are Networks Really Complex ?

Enterprise QOS Design & Deployment - Good, Bad Or Ugly ?

Focus on Your Business, Not Fancy Technologies

Are You Solving the Right Problem?

This Is What Makes Networking So Complex

Are Business Needs Just Excuses for Vendor Shenanigans?

The Three Paths of Enterprise IT

SDN Will Not Solve Real-Life Enterprise Problems

Why Intent Based Networking (IBN) will Not Save Your Network Anytime Soon ?

Complexity and the Thin Waist

It’s Most Complicated than You Think

Details and Complexity

How Many Routes My ASIC Can Hold ? - A Short Post

 

One of thing that always keep surprising me over the years is why 90% of pre-sales people in tech can't size the equipment correctly or don't have a solid logic to make those decisions, hence 99% of equipment are oversized to play safe and businesses have no clue that they are paying so much more. 

Obviously part of the problem is they have never come across formal training, course or book talking about it, even the most popular design and architecture books fails to address this or offer a comprehensive view/approach.

So instead of treating it as a science, most will resort to tactics.

In late 2021 I wrote this brief article around skills a network engineer should pick on in his/her early career. Where I suggested to be at least familiar with basic understanding of both "Router Architecture" & "ASIC Architecture". Now obviously the depth is always subjective to:

1. How many details I need to know for my current role & responsibilities in order to get things right

2. The amount of details and depth I need to know for potential future roles that you might be targeting 

3. Your personal curiosity & interest 

4. If you are really into Architecture & Design, You got to have fair & intermediate level of understanding of these topics at minimal

5. If you into Pre-sales, You often got to deal with sizing & performance for a given set of equipment as part of your deliverables requested by client in the form of RFP or RFI. Remember those data sheets you often have to refer to claiming IPv4 or IPv6 prefixes numbers a platform can hold/support ?

6. You might have to do platform testing at some point as part time or full time job including you may land yourself into a COE (Centre of Excellence) of your organization or may end up into a "Platform/Service Product Management" role.

Now you must know that often those specifics and details are hidden and never publicly shared/offered by most of ASIC/Platform vendors. You often got to be a premium customer and sign-off tons of NDAs to get those details to some extent and more importantly you got to be very specific around what exactly you are looking for since asking for data in abstraction would often result into tons of non-specific information thrown on you by your fav. ASIC vendor.

Assuming by now you have some more clarity in terms of why you need to know all these details as a Network engineer depending upon where you are and where you plan to end up, lets circle back to original topic for today which is "How many IPv4 (could be IPv6) prefix my device support in reality?"

Which leads us to a simple question - "What are the different variables I am dealing with when trying to figure how much routes my platform can really hold?"

While a simple answer you would often hear would be "it depends" or someone may point your to RFC-1925 rule 8 "It is more complicated than you think"

So let's try to list some of them in this series Part - 1

•  ASIC Architecture 
• ASIC pipeline
• Memory architecture
• Memory Carving/allocation to different features & functions (HW/SW)
• How the information is queued & de-queued 
• API details (Type of API, API interface, Information flow etc.)
• Routing Vs. Switching ASIC
• Hierarchical vs. A Flat FIB

•  NOS Architecture  
• How NOS is programming the FIB
• Prefix Length 
• Contiguous Vs. Dis-contiguous Prefixes
• Sorting Algorithm & Data Structures 
• Device Profiles/Resource Allocations by NOS 
• NOS Scheduler
• ECMP, UCMP, FRR

•  Platform Architecture
•  Line Card Architecture
•  Back Plane Architecture 

Further Readings:

ASIC

ASICs for Network Engineers

A summary of High Speed Ethernet ASICs

ASICs at the Edge

A Brief History of Router Architecture

Networking Chips versus GPUs/CPUs

Longest Prefix Matching in Networking Chips

Tearing Down the Memory Wall

Silicon Photonics and Integrated Optics

Need for Faster Innovation in EDA Tools for Design/Verification

Anatomy of Core Network Elements

SONiC: Open Source NOS in Data Cente

Cisco - Configuring SDM Resource Allocation Templates

Adjacency Matrix, Adjacency List, Priority Queue Implementation 

Juniper Networks Routing ASIC Strategy

Cisco 8000 Series - Under the Hood

Cisco NCS 5500 LPTS - iTCAM + eTCAM

Flat vs Hierarchical FIB

Sizing the Buffer

Sizing Router Buffers - Small is the New Big...

Embedded Hardware for Processing AI at the Edge: GPU, VPU, FPGA, and ASIC Explained

ASICs vs. Net Processors: Understanding the True Costs

P4 - Programming Protocol-Independent

Open Flow Specifications - Remember how Open Flow Originally planned to program the ASIC directly using Open Flow Controller ?

How Routers Really Work - A Webinar from Russ White under O'Reilly Subscription

Networking Hardware/Software Disaggregation in 2022

Select the Best Switching ASIC For the Job

Data Center Switching ASICs Tradeoffs

FIB Compression

Switching Hardware Series - Part 1 , Part 2 & Part 3

Juniper MX10000 LC480 Deepdive

BGP RIB Sharding

Using Trio -- Juniper Networks' Programmable Chipset -- for Emerging In-Network

Optimizing Power Consumption in High-End Routers

Striking a Balance: Exploring Fairness in Buffer Allocation and Packet Scheduling

Making 35 000 000 IP lookup operations per second with Patricia tree

Optimizing Power Consumption in High-End Routers

Saving Energy on PTX with PFE Power Off

ACX7000 L2 MAC Scale and Learning Rate

FIB Compression in Juniper Routers

PTX10001-36MR FIB Install Rate

Express 4 Filters - Foundation

Large Language Models — the hardware connection

Classification TCAM with Cisco CloudScale ASICs for Nexus 9000 Series Switches White Paper

Chiplets - The Inevitable Transition

[Podcast] The chips are down: Moore’s Law coming to an end

HTH...

A Network Artist 🎨

Network Automation Use Cases - A Mental Model

HTH...

A Network Artist 🎨

A Simple Go To Market (GTM) Model

 

HTH...

A Network Artist 🎨

The QOS Fallacies & Failures in a Modern Hybrid IT World - Part 1 of 2

 

When I wrote about QOS the last time around 7 years ago, I must say I had high hopes from SDN & IBN as both of the paradigms were still evolving. Meanwhile there have been some unsuccessful attempt to automate QOS by throwing some sort of controllers into the mix by few vendors beside some others claiming they can solve this problem with the mighty IBN.

As we are about to move into 2022, many still wonder if QOS makes any sense at all in the context of modern networking ?

In order to find the answers, let's break the problem into two parts:

1. Why QOS has been so unsuccessful historically

2. What are our options moving forward

So let's focus on point 1 to begin with.

1. How do we get started ? - Interestingly enough over a dozen books have been written on QOS over the last 2 decades or so in the context of IP networking which mostly talks about details such as congestion management vs. congestion avoidance and so forth. But very few of them actually jumps into platform specifics in terms of capabilities and dependencies (both HW & SW). More interestingly I personally haven't come across a single QOS book myself yet which gives you any practical advice or framework/methodology around how to gather technical requirements in reference to Applications in order to plan and craft a QOS policy. So often I have seen people struggling to come up with one and given most QOS deployments are tactical rather than strategic, people often have time constraints to come up with a one in a short time. That's why many times people end up coping some references from recommended design guides etc. which hardly works in real life (Unless you were too lucky !).

2. Benchmarking & Capacity Mgmt. - Most small, medium & even couple of the large enterprises that I have worked with including Telcos don't seem to have both of these as mature practices in place. Benchmarking is though one of the key exercises you need get through to craft a good QOS policy beside being a necessity in any of a mature Capacity mgmt. framework/practice. The other problem you may likely to run here is that in order to do effective benchmarking & capacity mgmt. you need to invest into additional visibility & performance mgmt. tools to gather the required details which are usually quite expensive beside that fact that you need to train your team on tools and required operating skills (for example statistical analysis, Time Series, Sampling details etc.). Certain times you are likely to run into the problem where in the given tool may not be able to offer you reporting/data that you need natively which means you are always dependent on tool vendor about if their product road-map is aligned to your priorities and timelines. And be careful about if the tool allows you to run custom reports or exports required data in format you need. So if your capacity mgmt. is still on excel sheets, you know where you are heading.

3. Measuring latency incorrectly - This is perhaps more common than you might have thought beside that fact that most QOS books don't offer any practical advice here too and details around how latency needs to broken down across the spectrum. Can you tell me the breakup of end to end latency (Server - Client) and that too on hop by hop basis ?

4. QOS Lifecycle Mgmt. - While this area has improved a bit when it comes to modern networking gear, assuming majority of equipment still out there are old ones which doesn't offer much when it comes to QOS lifecycle mgmt. that includes Plan, Design & Implementation. But more importantly what they lack are capabilities such as QOS monitoring & reporting in real time beside the correlation with network health & events. After all you don't want to hire someone today to type couple of show commands in every few minutes and running the scripts won't be that helpful either for most part when you are dealing with scale. Again there are couple of commercial and open source tools available but its an exercise which takes time and resources beside that fact you should know exactly what you are looking for in which scenario. BTW...will that resource be from planning team, tools team or ops team is what I leave for you to figure out in real life. :)

Also with the rise of modern solutions which are mostly built around the magical controllers and overlays, you must think about QOS FCAPS capabilities in such environment carefully. For example while from overlay protocol perspective everything might be just a single hop away, the packet eventually still gets passed through the physical world in underlay. So its importantly to find out early if QOS policies will be:

- Static or Dynamic in nature (Given you are using controller of some sort)

- Correlation of QOS statistics between underlay & overlay

- How policy gets propagated 

- How controller interacts with other systems and policies for introducing dynamic behavior 

- Does the system allows Time Based QOS policies (interestingly enough most don't yet)

- Dummy policy dry run capabilities if supported 

- How your QOS policy gels with your ISP agreements and how systems would talk to each other if at all depending upon SLA, Performance & Visibility/Reporting requirements both may agree upon

5. Policy Stitching - This is one of the hardest part to get across and more so in a multi-vendor environment. As mentioned earlier - beside the fact that most QOS books and vendor QOS courses don't cover much details around platform specifics and they just assume that one would figure it out, the things gets pretty complicated pretty quickly the moment you know that the QOS depends on:

- Platform and Specific Model you are using

- NOS version

- ASIC Architecture (ASIC Pipeline, Buffer, Memory type & speed, Over subscription, Queue/Dequeue algorithm etc.)

- Chassis specifics (in case you are using one as opposed to fixed form factor) - example VOQ, Switch Fabric, Fabric Generation, Fabric Modules Count etc...

- Supervisor Engine & Architecture beside its generation, CEF vs. dCEF kind of implementation specifics 

- Policy Framework supported by NOS - example hierarchical QOS, support for sub-interfaces/Logical interfaces, how policy aggregation works and in which direction etc.

6. Modern App. Architectures - Since these days some of the new buzz words in application space are Cloud Native Apps, Micro Services, Containers & Kubernetes etc. One might wonder how he/she would go about planning QOS for such environment which are highly dynamic in nature with complex topologies, both short & long lived flows with mix of interaction surfaces with other systems and tools such as distributed tracing to feedback into your QOS Mgmt. tool.

7. Complexity Induced by Networks - There are some very common network choices that every network architect makes at some point which further complicates the QOS implementation. These are perhaps some of those complexities which must exist in order to deliver the desired outcomes and are least avoidable such as:

- Snowflake Networks

- MC-LAG aka Port-Channels/Ether-Channels/Bundle Interfaces

- Multi-tenant Networks (Remember you only got few queues in HW)

- Dynamic Network Traffic Patterns (Even more so with TE Controllers) during stable conditions vs. failure conditions

- How TC gets implemented by a given vendor in given platform & NOS

- Inflated throughput & performance numbers by vendors (very common)

- Different SP QOS Models (Customer Facing vs. Core Facing) as they usually have no more than 3 bits or 8 classes to play around beside allocation models

- QOS in Dual Stack Networks vs. IPv6 only Networks

- Some nerd knobs such as QPPB

- Impact of Physical & Logical topology

Hope you find this helpful and lets continue with this in Part-2.

HTH...

A Network Artist 🎨