How Many Routes My ASIC Can Hold ? - A Short Post

 

One of thing that always keep surprising me over the years is why 90% of pre-sales people in tech can't size the equipment correctly or don't have a solid logic to make those decisions, hence 99% of equipment are oversized to play safe and businesses have no clue that they are paying so much more. 

Obviously part of the problem is they have never come across formal training, course or book talking about it, even the most popular design and architecture books fails to address this or offer a comprehensive view/approach.

So instead of treating it as a science, most will resort to tactics.

In late 2021 I wrote this brief article around skills a network engineer should pick on in his/her early career. Where I suggested to be at least familiar with basic understanding of both "Router Architecture" & "ASIC Architecture". Now obviously the depth is always subjective to:

1. How many details I need to know for my current role & responsibilities in order to get things right

2. The amount of details and depth I need to know for potential future roles that you might be targeting 

3. Your personal curiosity & interest 

4. If you are really into Architecture & Design, You got to have fair & intermediate level of understanding of these topics at minimal

5. If you into Pre-sales, You often got to deal with sizing & performance for a given set of equipment as part of your deliverables requested by client in the form of RFP or RFI. Remember those data sheets you often have to refer to claiming IPv4 or IPv6 prefixes numbers a platform can hold/support ?

6. You might have to do platform testing at some point as part time or full time job including you may land yourself into a COE (Centre of Excellence) of your organization or may end up into a "Platform/Service Product Management" role.

Now you must know that often those specifics and details are hidden and never publicly shared/offered by most of ASIC/Platform vendors. You often got to be a premium customer and sign-off tons of NDAs to get those details to some extent and more importantly you got to be very specific around what exactly you are looking for since asking for data in abstraction would often result into tons of non-specific information thrown on you by your fav. ASIC vendor.

Assuming by now you have some more clarity in terms of why you need to know all these details as a Network engineer depending upon where you are and where you plan to end up, lets circle back to original topic for today which is "How many IPv4 (could be IPv6) prefix my device support in reality?"

Which leads us to a simple question - "What are the different variables I am dealing with when trying to figure how much routes my platform can really hold?"

While a simple answer you would often hear would be "it depends" or someone may point your to RFC-1925 rule 8 "It is more complicated than you think"

So let's try to list some of them in this series Part - 1

•  ASIC Architecture 
• ASIC pipeline
• Memory architecture
• Memory Carving/allocation to different features & functions (HW/SW)
• How the information is queued & de-queued 
• API details (Type of API, API interface, Information flow etc.)
• Routing Vs. Switching ASIC
• Hierarchical vs. A Flat FIB

•  NOS Architecture  
• How NOS is programming the FIB
• Prefix Length 
• Contiguous Vs. Dis-contiguous Prefixes
• Sorting Algorithm & Data Structures 
• Device Profiles/Resource Allocations by NOS 
• NOS Scheduler
• ECMP, UCMP, FRR

•  Platform Architecture
•  Line Card Architecture
•  Back Plane Architecture 

Further Readings:

ASIC

ASICs for Network Engineers

A summary of High Speed Ethernet ASICs

ASICs at the Edge

A Brief History of Router Architecture

Networking Chips versus GPUs/CPUs

Longest Prefix Matching in Networking Chips

Tearing Down the Memory Wall

Silicon Photonics and Integrated Optics

Need for Faster Innovation in EDA Tools for Design/Verification

Anatomy of Core Network Elements

SONiC: Open Source NOS in Data Cente

Cisco - Configuring SDM Resource Allocation Templates

Adjacency Matrix, Adjacency List, Priority Queue Implementation 

Juniper Networks Routing ASIC Strategy

Cisco 8000 Series - Under the Hood

Cisco NCS 5500 LPTS - iTCAM + eTCAM

Flat vs Hierarchical FIB

Sizing the Buffer

Sizing Router Buffers - Small is the New Big...

Embedded Hardware for Processing AI at the Edge: GPU, VPU, FPGA, and ASIC Explained

ASICs vs. Net Processors: Understanding the True Costs

P4 - Programming Protocol-Independent

Open Flow Specifications - Remember how Open Flow Originally planned to program the ASIC directly using Open Flow Controller ?

How Routers Really Work - A Webinar from Russ White under O'Reilly Subscription

Networking Hardware/Software Disaggregation in 2022

Select the Best Switching ASIC For the Job

Data Center Switching ASICs Tradeoffs

FIB Compression

Switching Hardware Series - Part 1 , Part 2 & Part 3

Juniper MX10000 LC480 Deepdive

BGP RIB Sharding

Using Trio -- Juniper Networks' Programmable Chipset -- for Emerging In-Network

Optimizing Power Consumption in High-End Routers

Striking a Balance: Exploring Fairness in Buffer Allocation and Packet Scheduling

Making 35 000 000 IP lookup operations per second with Patricia tree

Optimizing Power Consumption in High-End Routers

Saving Energy on PTX with PFE Power Off

ACX7000 L2 MAC Scale and Learning Rate

FIB Compression in Juniper Routers

PTX10001-36MR FIB Install Rate

Express 4 Filters - Foundation

Large Language Models — the hardware connection

Classification TCAM with Cisco CloudScale ASICs for Nexus 9000 Series Switches White Paper

Chiplets - The Inevitable Transition

[Podcast] The chips are down: Moore’s Law coming to an end

HTH...

A Network Artist 🎨

Network Automation Use Cases - A Mental Model

HTH...

A Network Artist 🎨

A Simple Go To Market (GTM) Model

 

HTH...

A Network Artist 🎨

The QOS Fallacies & Failures in a Modern Hybrid IT World - Part 1 of 2

 

When I wrote about QOS the last time around 7 years ago, I must say I had high hopes from SDN & IBN as both of the paradigms were still evolving. Meanwhile there have been some unsuccessful attempt to automate QOS by throwing some sort of controllers into the mix by few vendors beside some others claiming they can solve this problem with the mighty IBN.

As we are about to move into 2022, many still wonder if QOS makes any sense at all in the context of modern networking ?

In order to find the answers, let's break the problem into two parts:

1. Why QOS has been so unsuccessful historically

2. What are our options moving forward

So let's focus on point 1 to begin with.

1. How do we get started ? - Interestingly enough over a dozen books have been written on QOS over the last 2 decades or so in the context of IP networking which mostly talks about details such as congestion management vs. congestion avoidance and so forth. But very few of them actually jumps into platform specifics in terms of capabilities and dependencies (both HW & SW). More interestingly I personally haven't come across a single QOS book myself yet which gives you any practical advice or framework/methodology around how to gather technical requirements in reference to Applications in order to plan and craft a QOS policy. So often I have seen people struggling to come up with one and given most QOS deployments are tactical rather than strategic, people often have time constraints to come up with a one in a short time. That's why many times people end up coping some references from recommended design guides etc. which hardly works in real life (Unless you were too lucky !).

2. Benchmarking & Capacity Mgmt. - Most small, medium & even couple of the large enterprises that I have worked with including Telcos don't seem to have both of these as mature practices in place. Benchmarking is though one of the key exercises you need get through to craft a good QOS policy beside being a necessity in any of a mature Capacity mgmt. framework/practice. The other problem you may likely to run here is that in order to do effective benchmarking & capacity mgmt. you need to invest into additional visibility & performance mgmt. tools to gather the required details which are usually quite expensive beside that fact that you need to train your team on tools and required operating skills (for example statistical analysis, Time Series, Sampling details etc.). Certain times you are likely to run into the problem where in the given tool may not be able to offer you reporting/data that you need natively which means you are always dependent on tool vendor about if their product road-map is aligned to your priorities and timelines. And be careful about if the tool allows you to run custom reports or exports required data in format you need. So if your capacity mgmt. is still on excel sheets, you know where you are heading.

3. Measuring latency incorrectly - This is perhaps more common than you might have thought beside that fact that most QOS books don't offer any practical advice here too and details around how latency needs to broken down across the spectrum. Can you tell me the breakup of end to end latency (Server - Client) and that too on hop by hop basis ?

4. QOS Lifecycle Mgmt. - While this area has improved a bit when it comes to modern networking gear, assuming majority of equipment still out there are old ones which doesn't offer much when it comes to QOS lifecycle mgmt. that includes Plan, Design & Implementation. But more importantly what they lack are capabilities such as QOS monitoring & reporting in real time beside the correlation with network health & events. After all you don't want to hire someone today to type couple of show commands in every few minutes and running the scripts won't be that helpful either for most part when you are dealing with scale. Again there are couple of commercial and open source tools available but its an exercise which takes time and resources beside that fact you should know exactly what you are looking for in which scenario. BTW...will that resource be from planning team, tools team or ops team is what I leave for you to figure out in real life. :)

Also with the rise of modern solutions which are mostly built around the magical controllers and overlays, you must think about QOS FCAPS capabilities in such environment carefully. For example while from overlay protocol perspective everything might be just a single hop away, the packet eventually still gets passed through the physical world in underlay. So its importantly to find out early if QOS policies will be:

- Static or Dynamic in nature (Given you are using controller of some sort)

- Correlation of QOS statistics between underlay & overlay

- How policy gets propagated 

- How controller interacts with other systems and policies for introducing dynamic behavior 

- Does the system allows Time Based QOS policies (interestingly enough most don't yet)

- Dummy policy dry run capabilities if supported 

- How your QOS policy gels with your ISP agreements and how systems would talk to each other if at all depending upon SLA, Performance & Visibility/Reporting requirements both may agree upon

5. Policy Stitching - This is one of the hardest part to get across and more so in a multi-vendor environment. As mentioned earlier - beside the fact that most QOS books and vendor QOS courses don't cover much details around platform specifics and they just assume that one would figure it out, the things gets pretty complicated pretty quickly the moment you know that the QOS depends on:

- Platform and Specific Model you are using

- NOS version

- ASIC Architecture (ASIC Pipeline, Buffer, Memory type & speed, Over subscription, Queue/Dequeue algorithm etc.)

- Chassis specifics (in case you are using one as opposed to fixed form factor) - example VOQ, Switch Fabric, Fabric Generation, Fabric Modules Count etc...

- Supervisor Engine & Architecture beside its generation, CEF vs. dCEF kind of implementation specifics 

- Policy Framework supported by NOS - example hierarchical QOS, support for sub-interfaces/Logical interfaces, how policy aggregation works and in which direction etc.

6. Modern App. Architectures - Since these days some of the new buzz words in application space are Cloud Native Apps, Micro Services, Containers & Kubernetes etc. One might wonder how he/she would go about planning QOS for such environment which are highly dynamic in nature with complex topologies, both short & long lived flows with mix of interaction surfaces with other systems and tools such as distributed tracing to feedback into your QOS Mgmt. tool.

7. Complexity Induced by Networks - There are some very common network choices that every network architect makes at some point which further complicates the QOS implementation. These are perhaps some of those complexities which must exist in order to deliver the desired outcomes and are least avoidable such as:

- Snowflake Networks

- MC-LAG aka Port-Channels/Ether-Channels/Bundle Interfaces

- Multi-tenant Networks (Remember you only got few queues in HW)

- Dynamic Network Traffic Patterns (Even more so with TE Controllers) during stable conditions vs. failure conditions

- How TC gets implemented by a given vendor in given platform & NOS

- Inflated throughput & performance numbers by vendors (very common)

- Different SP QOS Models (Customer Facing vs. Core Facing) as they usually have no more than 3 bits or 8 classes to play around beside allocation models

- QOS in Dual Stack Networks vs. IPv6 only Networks

- Some nerd knobs such as QPPB

- Impact of Physical & Logical topology

Hope you find this helpful and lets continue with this in Part-2.

HTH...

A Network Artist 🎨

A Simple Routing Protocols Decomposition Model - Part 2 (Peering Mgmt.)

 

So in the first part of the series, we started with rather a simple decomposition model to get bit more insights into the routing protocol internals.

So let's continue the series by expanding on the very first layer in our model - Peering Management.

In any routing protocol before we get fancy in terms of which all features, functions and knobs to use, the very basic requirement is to peer with other devices into the network since eventually a routing protocol is nothing but a distributed database. Routing protocols are designed to convey different set of information to its peer devices such as:

- Topology Information

- Reachability Information

- Policy Information

The type of information that a given routing protocol would exchange with its peers would largely depend on the protocol itself (OSPF, IS-IS, BGP) & where that proposed routing protocol is used into the network (Campus, WAN, Metro-E, DC) as implementation specifics do change.

[ Click on Image to Enlarge ]

As you may notice, there are lot of things working behind the scenes when it comes to peering in a routing protocol context. But don't get carried away by looking at the complexity. Once you look closely, all of these pieces kind of makes sense.

So let's start with the top row, reading it from left to right. 

Self Identity - Before the routing protocol determines whom it needs to communicate with and what information needs to be exchanged, it must find its own identity first. The most common way to give an identity to the routing protocol instance itself is assigning it a router id (RID). The RID can be configured manually or it can be derived automatically depending upon the platform and NOS. 

In real life assuming network virtualization is much more common today, you are allowed to configure unique RID for each routing protocol as well as a unique RID for each instance/process under same protocol. 

Protocol Addressing - Once the routing protocol is able to define its identify with a RID, the next step is to understand it's addressing. The addressing serves many purposes but the most basic one is to provide location services from the overall network view standpoint. Though protocols such as LISP was an attempt to separate device identity from device location, due to limited use cases (such as mobility) and other problems it never really took off well.

Every routing protocol has it's own addressing scheme which may further have impact on its scaling and expected working behavior if not done correctly.

Participation - The next step for the routing protocol is to determine its participating interfaces on a given device and in certain cases the entire device itself. Depending upon the design you may run into some interesting challenges though.

Reliable Transport - Every routing protocol needs a reliable transport to be able to effectively communicate with its peer device. The reliability itself is an important aspect and while some routing protocols such as BGP rely upon existing TCP stack, others like OSPF uses IP protocol 89 & EIGRP choose its own transport protocol namely RTP.

Neighbor Discovery - In the next step the protocol must discover its peer/neighbor/adjacent device depending upon which routing protocol you are following. The protocol while ideally should keep a track of its neighbor and relationship state, this may or may not be implemented.

The neighbor could be configured manually or dynamically discovered, while the discovery phase itself may use unicast or multicast as a transport for reachability purpose to the next hop device. Also the reachability to the neighbor could be over layer 2 transport or layer 3 transport depending upon the protocol. IS-IS and many other IOT industrial protocols operate at layer 2 for example. In case of eBGP, the neighbor in fact may be multiple physical hops away.

Neighbor Identity - While we may have discovered our neighbor, it doesn't mean the neighbor itself is a an intended neighbor or a legitimate neighbor always. After all someone might want to spoof or sometimes we may end up discovering somebody completely un-intentionally. So sharing any information with an unexpected neighbor won't make any sense. To prevent this we have several measures which we can put in place such as Authentication, Validating neighbor's identity (remember they also have RID), Validating neighbor based on IP Packet's TTL value etc. such as in OSPF & BGP. The modern day solutions such as SD-WAN usually uses RPKI over TLS/DTLS channel.

Establish Session - Once the neighbor is discovered and validated, we finally establish a session with it. Depending upon the protocol, we might have a single session vs. multiple sessions going on. A simple example would be networks running IPv4 & IPv6 at the same time under single routing protocol instance. While some implementations exchange information related to both IPv4 and IPv6 over a single session, some may do it over a separate dedicated session for each of them. Long time ago there was an attempt to run multi session bgp for MTR (Multi Topology Routing) for network virtualization use cases.

Capabilities Exchange -  This is an another interesting step where the routing protocol running on separate devices exchange capabilities with each other to find the lowest common denominator. For example we know BGP is more like an application that runs on top of TCP as opposed to a pure layer 3 routing protocol which only carries routes. BGP though can carry layer 3 routing information and in case with most vendors it's been the default behavior, BGP does allow us to carry many other set of information depending upon the use case in the form of AFI/SAFI which is essentially an encoding format. For example BGP can carry MAC Addresses information under Layer 2 VPN EVPN address family when enabled. Though with BGP, you got to be cautious about enabling a new capability/address family in production network as highlighted here.

Establish Adjacency - The protocol reach this far and finally the peers are ready to exchange required set of information needed to populate RIB & other details such as topology graph. An interesting example in a routing protocol context would be (In case you are wondering in which scenario two devices would be neighbors but not adjacent) OSPF.

Messages Exchange - Finally we [assuming by now you are thinking like a routing protocol :) ] reach this stage where in we finally start exchanging information through messages. The messages needs to be sent reliably, keeping track of to understand which one to prefer in case of being received from multiple sources, acknowledged and so forth besides how to queue and dequeue them and at what intervals those should be sent out vs. being hold back for a while to pack multiple events together for optimization and getting the latest information being sent out.

Further Readings:

Cisco IP Routing: Packet Forwarding and Intra-domain Routing Protocols: Packet Forwarding and Intra-domain Routing Protocols 

Network Routing: Algorithms, Protocols, and Architectures

Network Algorithmics,: An Interdisciplinary Approach to Designing Fast Networked Devices

Inside Cisco Ios Software Architecture

HTH...

A Network Artist 🎨