State of Networking Industry In 2021... a bit of sarcasm with a pinch of salt :)

 
- Network Engineers those are still using "CLI" are "CLI junkies" & "dated"

- Network Engineers those switched to GUI in the holy name of SDN/IBN Solutions are " cool "

- Network Engineers those are often writing scripts in YAML & JINJA are "coolest"....shall we call it "The Holy CLI Mode" ?

- Network Engineers using "Machine Learning" are " Gods of the thunder"

And now You can safely forget about - Robust Network Design & Implementation, Standardization, Modularity & Failure domains, Statistical Analysis, People-Processes-Tools

HTH...

A Network Artist 🎨

Facebook Down Event - The dilemma of a CTO, Black Swans & Fallacies of IBN

While Facebook just seem to have published somewhat a lengthysh version of root cause analysis (https://lnkd.in/guptSB3u) for public about their recent worldwide network outage that made - Facebook, WhatsApp & Instagram completely cut out from internet, it must have raised some concerns in the worldwide CTO and CIO community.

Since historically they have been told that and what pretty much every vendor in networking industry is preaching about in terms of different ways & methods (Systems, People & Processes) to avoid such circumstances are bright & magical ideas such as:

- Automation & Orchestration
- Intent Based Networking (IBN)
- Software Defined Networking (SDN)
- Centralized Controllers
- Data Models
- Automated Test & Deployment Pipeline with Unit Tests (aka CICD)
- Reliability & Resiliency Engineering
- AI/ML OPS
- Network Design Principles (Hierarchy, Swim Lanes, Segmentation etc.)
- Streaming Telemetry
- Observability Tools
- Bright Engineers
- Testbed Equipment
- Network Modelling & Simulation Tools with Formal Verification
- Rigorous platforms testing (HW/SW)
- Single Source of Truth
- Chaos Engineering
- BCP Plan
- Correlation Tools & what not

But assuming if you go via this checklist, Facebook would probably have all checks against all these items and so would be any of the FAANG company at this stage.

" So assuming you are a CTO or CIO, what would you suggest as possible next steps to your CEO & board if you have been called up this week for a meeting to discuss about how do we ensure such events don't happen in our network ? "

So lets park the above question for a while and move to what reactions we have seen so far.

1. The usual suspect is, bad things happens and everything breaks at some point, focus on RCA...move on and ensure it doesn't happen again

2. Network Architects favorite answer.... " it depends "

3. Was it a People or Process issue ?

4. The conspiracy theory that FB was under a Cyber attack which they don't want to disclose

5. Blame BGP (the easy suspect) … interestingly we got 10000+ new BGP experts on twitter and LinkedIn overnight :) beside the fact that 99% of them hardly understand the BGP details since none of them looked at the problem from perspectives of "unintended consequences", "ripple effect", "interaction surfaces", "failure domains", " & so forth beside all the pointers list I shared above. So let's say blaming BGP was an easy pick for the "ghost" network engineers. Beside the fact that RCA published by Facebook doesn't cover any technical details either.

6. "The Black Swans" - This is an interesting one and less talked about fact in case of this outage. While some may claim this was just one of those black swan events, I personally seriously doubt that and more so in the absence of a detailed RCA.

Further Readings:

Helping our customers through the CrowdStrike outage

RFC 1925 - The Twelve Networking Truths

RFC 3439 - Some Internet Architectural Guidelines and Philosophy

RFC 1149 - A Standard for the Transmission of IP Datagrams on Avian Carriers

The Law of Leaky Abstractions

Grey Failure Lessons Learned

Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems

How Complex Systems Fail

Nonlinear Effects of Optimization-Induced Complexity

Complexity and the Thin Waist

Seamless Suffering

Why Does Internet Keep Breaking?

Paxos

Raft

A Byzantine failure in the real world

Decentralizing the Internet, Again

The Decentralized Internet: Rethinking Trust

HTH...

A Network Artist 🎨

The Reality is Not As You See It - Network Engineering Pro Tips

In Enterprise Networking context, 90% probability is that You are never going to encounter " Multi Area OSPF ", " OSPF Virtual Links ", " OSPF as PE-CE Routing Protocol ", " EIGRP as PE-CE Routing Protocol " & " BGP Confederations ", assuming " RIP " is by now resting in peace ✌

Unless...

1. You want to be fancy
2. You are preparing for your favorite vendor's expert level lab exam
3. Your network design is broken
4. Perhaps you have really good and convincing reasons to opt. for odds

And of course its worth knowing - how anti-patterns such as SD-WAN and MPLS Layer-3 VPNs makes it even rare besides tons of overlays you see around.

So next time you see those on your favorite Networking vendor exam blueprint, don't shy away to ask ...Why ?

HTH...

A Network Artist 🎨

Why Intent Based Networking (IBN) will Not Save Your Network Anytime Soon ?

 

In every few years, your favorite networking equipment vendor/provider/manufacturer comes to you with something new, suggesting this is the thing you need to solve all your networking problems in order to deliver the " Business Outcomes " that your CxO team is most desperate to achieve since very next second after the " Big Bang ".

Over the years both Networking Industry & Networking OEMs keep coming with new ideas 💡 - Separation of Planes, Centralized life cycle management, BGP as the mothership protocol by fitting every possible thing within that, Separating Policy from - Topology Information & Reachability Information, The magical 3 letters word " SDN ", Policy Based Networking, Application Aware Networking, Automated remediation and last but not least " The Magical Intent Based Networking aka IBN " beside AI Ops and what not.

But, You often forgot to involve qualified business people in defining those modern networking standards and terms, and for whatever reasons neither side ever took interest to step into the other side of the territory to get its basics right or to discuss what's really needed.

So the choice you are left with at the movement is - either you start taking those small steps or wait for IBN to fail (Since SDN and Open Flow have been declared dead already) on delivering those " Business Outcomes " and the cycle keeps on repeating endlessly.

And in case you doubt this, remember:

1. The intent based networking draft doesn't define or talk about " Business Outcomes " or even " Business Intent ".

2. Crazy Network Automation nerds should go and read millions of lines of code that a lady wrote for Apollo 11, which A. Tells you there is fundamentally nothing new in doing that and B. You can only automate better if You truly know the science behind stuff that you are trying to automate but more importantly have been able to frame the problem well enough. But let's circle back on those details and why often network automation fails to deliver true " Business Outcomes " for later post.

3. Most Network Engineers are not on party list when it comes to CxO meetings and most CxOs are either not invited for Network Strategy and Planning Discussions. Beside some exceptions where either party is there to have a cup of good coffee ☕

So you either start living with those ideas and keep chasing the unicorns, or you can take a step back and start thinking how to really get it right...or let's have a chat 😉 since you must remember that living under a rock is a choice...

HTH...

A Network Artist 🎨

What Digital Transformation is actually " Not " about

- Technology Adoption
- Technology Lead Innovation
- Customer Experience/CX
- User Experience/UX
- Creating Value / Value Proposition / Value Stream Mapping
- Leveraging Data / Dat Science / AI / ML / Analytics
- Competition or Coopetition
- Digital Strategy
- Business Model
- Process Mgmt. / Process Optimization
- Platforms & Eco-Systems
- Testing Business Ideas Fast & Cheap
- Simplicity

All of those are fallacies :) , didn't get anyone there, won't get you there … all of those are " Empty Words " & " Consequences " 

HTH...

A Network Artist 🎨