Optical Transceivers Cost You More Than You Think

In the world of high-performance networking, engineers and operators often focus their attention on the switch chassis itself. When evaluating power budgets or structural installation, it is common to view the switch as the primary, expensive "brain" of the system, while treating the SFP transceiver modules as mere auxiliary accessories—side components that add little to the overall system load. However, a closer analysis of high-density systems, such as the 48-port Cisco Catalyst 9500, proves that this assumption is a significant misconception.

This article examines the validity of this assumption. Let us look closer at all transceivers versus the switch main body.

Cisco Catalyst 9500 (C9500-48Y4C)
Source (Thanks!):
Cisco Catalyst 9500 Series Switches Hardware Installation Guide

25G transceiver (SFP-25G-SR-S)
Source (Thanks!):
Cisco 25GBASE SFP28 Modules Data Sheet

I choose Cisco Catalyst 9500 (C9500-48Y4C) as an example target for analysis. This Catalyst 9500 has forty-eight user ports, and I install forty-eight 25G transceivers (SFP-25G-SR-S) into each port. We analyze in three dimensions: Power Consumption, Heat Generation, and Financial Cost.

MPO/MTP Fiber Optic Connectors

To achieve higher transmission bit rate, Ethernet fiber optic transceivers often utilize parallel optics to transmit data across multiple fiber lanes simultaneously.

Think of a single pair of fibers as one data lane for sending and receiving. By bundling four pairs of fibers, a transceiver can transmit and receive four times the amount of data simultaneously. For instance, if a single fiber pair delivers a throughput of 10 Gbps, a four-pair configuration can scale up that speed fourfold to 40 Gbps.

Standard transceivers that only require a single pair of fibers typically use LC (Lucent Connector) interfaces. However, when a transceiver needs to connect multiple fiber pairs through a single port, it must use high-density MPO/MTP connectors.

MPO (Multi-fiber Push-On)

MPO is an open industry standard. Its name perfectly reflects its design philosophy: a high-density, multi-fiber connector with a simple push-on mating mechanism. Common configurations include MPO-8 (8 parallel fibers) and the more widespread MPO-12 (12 parallel fibers).

Figure 1: MPO-12 vs MPO-8
Source (Thanks!):
https://www.flukenetworks.com/blog/cabling-chronicles/getting-12-and-8-fiber-polarity-right

Cool Immersive Cooling

I was so amazed about this cooling technology. It is new to me.

Thinking about liquid cooling for servers and networking devices, coming into my mind are tightly sealed water pipes around the circuit boards. This is different. The circuit board and everything simply immerse in liquid.

Cisco Service Provider Tweet

Three possible scenarios of Software Defined Networking (SDN)

Software-defined networking (SDN) is an approach to create a centrally controlled programmable packet network. Any protocols with the same approach could be considered as SDN as well.

For open protocols, we have one popular standard protocol “OpenFlow” talking among the central controllers to all managed networking devices. Open Network Foundation (ONF) defines OpenFlow protocol.

In fact, vendors also have developed proprietary protocols to implement this same approach. For example, Cisco’s ACI is a proprietary SDN solution.

Here I summarize 3 most probable scenarios when we deploy SDN.

Clear configured allowed VSAN list on trunk ports of Cisco MDS

A friend asked a good question about how this command works on Cisco MDS FibreChannel switch: “switchport trunk allowed vsan all”.

To my surprise, I cannot find any specific official documents to explain it clearly. I did some experiment on one Cisco MDS 9148. And here is my conclusion.

Bay, beach, and cliff near Chung-De Station. (崇德海灣).
Hualian County, Taiwan.

Video @Scale, "Introducing 6-pack, the First Open Hardware Modular Switch"

I really should have found and watched this video earlier!

This video is presented by Yuval Bachar, the same person posted this announcement on the Facebook Blog.

Here are some quick notes I took after watching this video.

Two quick observations of Facebook’s announcement of “6-pack” open hardware modular switch

Facebook announced an introduction to its "6-pack" open hardware module switch 20 hours ago. Here are my two quick observations of "6-pack".

Three quick things I have learned from Facebook’s Next Generation Data Center Network

Facebook published this post about their next-generation Data Center Network. Here I am noting down three quick things I have learned from it.

A view to the Pacific Ocean inside Farglory Ocean Park, Hualian County, Taiwan

New product: Cisco UCS "Mini"

Cisco has a new update few days ago to her Unified Computing System (UCS): the UCS Mini. Here is what I have known of it from the public web sites.

Sun is rising at Hualian County of Taiwan on August 30, 2014.

Threatening the market by “Wedge”? I don't think so.

A grass next to Taipei Botanical Garden (台北植物園), Taiwan

I just don't get it. Products such as iPhone/iPad are totally proprietary. And people still love them. Network devices are also proprietary. However, people just don't like proprietary network boxes and want them all to become as open as commodities?

Last Wednesday (June 18, 2014) Facebook announced the existence of their experimental in-house Top of Rack 40Gbps switch called “Wedge”. The basic ideas in many reports from the press (like this, this, and this) are: this would threaten existing market of network vendors such as Cisco.

Bandwidth (Throughput) is not everything about network performance

A look along the Coconut Palm Avenue (椰林大道).
National Taiwan University.

We like to compare the "bandwidth" numbers of different network systems. In my experience, bandwidth is the most asked property when I am introducing new technologies to other network workers. Bandwidth is very easy for people to understand, to do the calculations, and to compare which one is better than the others.

However, bandwidth is just not everything. Network with larger bandwidth is simply not a faster network at all.

Here I crafted an imaginary example, to show you why bandwidth is not as important as you might think of.

This imaginary example is called: "Trans-Pacific High Bandwidth Pipe".

Northbound, Southbound, and East/Westbound. What do they mean?

We hear a lot of directions when we are talking about Data Center technologies: Northbound, Southbound, and even Eastbound/Westbound. What do they mean? Why are they called this way?

Software Defined Networking (SDN)

Source: Wikipedia

For minimum discussion of SDN, three layers are defined inside an SDN. They are SDN Service Layer, Controller, and Physical Network.

Notes of PONG command on Nexus 7000

Mountain view over Devine Trees near Chilan (棲蘭), Yilan County, Taiwan. (棲蘭神木園)

We can use PONG command to measure and and analyze switch port to other switch port latency. Example of using this command can be found on PONG command reference.

Cisco NX-OS vPC (Virtual PortChannel) Configuration Notes

To enable Multi-chassis EtherChannel (MEC) on Cisco Nexus data center switches, the only option left for you is to enable Virtual PortChannel (vPC).

Virtual PortChannel is not that easy for people to configure correctly by hand. Even In this “Quick” vPC Configuration Guide, we still have to do 8 long steps before we have some working vPCs. A little bit frustrating when we look at it for the first time.

Here I just want to point out one aspect to make you feel better about vPC.

Displaying text file contents on Cisco IOS and NX-OS

[Cisco IOS]

For Cisco IOS platforms, we use the command more.
http://www.cisco.com/en/US/docs/ios/fundamentals/command/reference/cf_m1.html#wp1022989

Enable Jumbo Frame function of Cisco Nexus 5000 and 7000

An autumn view of Taipei Campus of National Taipei University (NTPU).

For Data Center networking, and in most of the cases, jumbo frames should be enabled. However, Jumbo frame feature is not enabled by default. So the next question would be "How to enable them in Cisco's Data Center switches such as Nexus 5000 and Nexus 7000 families"?

The answer is in this page on Cisco's official site.

No more CMP on Nexus 7000 next-generation supervisor modules Sup2 and Sup2E

Connectivity Management Processor (CMP) is like an embedded terminal server dedicated for a single Nexus 7000 data center switch. I personally like this feature because we do not have to prepare another set of terminal server just to monitor the console output while we reboot the switch.

However, starting from Supervisor 2 (Sup2) and Supervisor 2 Enhanced Module (Sup2E), CMP hardware is removed from supervisor module itself.

I am not sure whether this feature will come back in the future or not. For the time being, we will have to prepare terminal servers or reserve asynchronous serial ports for Nexus 7000 switches administration.


[Supervisor 1]

Captured on Cisco.com.

1	CMP Status LED	4	ACT LED
2	Link LED	5	CMP MGMT Ethernet LED
3	CMP MGMT Ethernet port

1

CMP Status LED

4

ACT LED

2

Link LED

5

CMP MGMT Ethernet LED

3

CMP MGMT Ethernet port

Cisco UCS C210 in Greenplum MR Hadoop Reference Configuration

Major hardware components and interconnection of Greenplum MR Hadoop Reference Configuration. This diagram is a screen capture of Figure 5 of  this White Paper on Cisco.com.

In fact, I shared the following content in this Google Plus Community: Using Cisco UCS. If you like to explore more about Cisco UCS, welcome to join us anytime!

---

I used to say UCS C Series are designed for "smaller deployment". However, recently I found an interesting application in DCUCD v5.0 training materials,  which contradicts to what I said before. Greenplum MR Hadoop Reference Configuration is a quite-large deployment with UCS C Series only. I really should correct my saying from now on!

My study notes on Cisco ASA 1000V video: "See the Cisco ASA 1000V in Action"

Here are the points I learned from this video. Have fun with it!

Avoiding human errors with Cisco NX-OS "Grace Periods"

Looking out to Guishan Island (the left top far)
Taken around Northeast Coast National Scenic Area (東北角海岸國家風景區), Yilan County, Taiwan.

Cisco offers “Evaluation” time intervals (Grace Periods) for some features of Cisco NX-OS on Data Center switches such as Nexus 7000 or Nexus 5000. It is a nice design because customer can evaluate whether those specific features are suitable for their needs or not, before they really buy that license. This is also very useful for PoC (Proof of Concept) installation.

It is really simple to start evaluation. First we enable evaluation grace periods by “license grace-period” command. Then we only have to type “feature X” commands at the CLI as if we have bought the license.