Three quick things I have learned from Facebook’s Next Generation Data Center Network

Facebook published this post about their next-generation Data Center Network. Here I am noting down three quick things I have learned from it.

A view to the Pacific Ocean inside Farglory Ocean Park, Hualian County, Taiwan

East-west bound traffic is growing much faster than North-South bound traffic for applications like Facebook.

In this post, Alexey mentioned “machine to machine” traffic and “machine to user” traffic. I believe “machine to machine” is exactly East-West bound and “machine to user” is exactly North-South bound traffic. A video explaining the growth of “machine to machine” traffic is also included in the original post.

(See 0'39" for the growth comparison between "machine to machine" and "machine to user" traffic)

Design requirements for DC network today for switch to switch ports should start from 40Gbps, and server to switch port should start from 10Gbps.

In this architecture, all ports among the switches are 40Gbps. All Top-of-Rack (TOR) switches to servers are 10Gbps. I believe this should be the starting point for any new Data Center Networks today.

Diagram Captured on Facebook Code Blog post.
Original title is "Figure 2: Schematic of Facebook data center fabric network topology"

Layer 3 server to server design is working well for applications like Facebook.

For this design, the maximum server to server IP hops could be: TOR -> Fabric Switch -> Spine Switch -> Fabric Switch -> TOR. That is 5 router hops between two servers.


One more thing…

Facebook announced in June this year about their experimental 16x 40Gbps-port Wedge switch. I also discussed about it at that time. Although it is not mentioned inside this post at all, I can only guess each single Fabric Switch and Spine Switch could be composed of 6 Wedge switches. Since all of the network devices are running in Layer 3, it really does not matter whether one single switch is really running as one, or six individual switches.

Of course, this is only my guessing. Please correct me if you know Facebook releases any new facts in the future!


References:

Introducing data center fabric, the next-generation Facebook data center network | Engineering Blog | Facebook Code