Facebook’s new Iowa data center goes modular to grow forever

By Stephen Lawson | Nov 14, 2014

The traffic inside Facebook’s data centers is growing so fast that the company is changing the basic architecture of its networks in order to keep up.

The new design, which Facebook calls a data center fabric, doesn’t use a new link speed to make the network run faster. Instead, it turns the whole system into a set of modules that are less expensive and more widely available than what the company’s using now. It’s also easier to deploy and manage, according to Facebook’s networking chief.

Unlike older hierarchical networks, the modular design can provide consistently fast links across the data center for any two servers to talk to each other. The new architecture was used in a 476,000-square-foot (44,000-square-meter) data center that goes online today in Altoona, Iowa. Facebook plans to use it in all newly built centers and retrofit older facilities as part of its regular upgrade cycle.

Facebook and other companies with sprawling Internet data centers have turned to homegrown or inexpensive white-box gear for networking as well as for computing. They add their own software on top of that hardware, which can mean they don’t buy dedicated products from networking specialists such as Cisco Systems. Though most enterprises don’t have the network scale or in-house expertise to do the same, software-defined technologies developed and shared by these trailblazers are changing some aspects of networking.

Facebook’s current data-center networks are based on clusters, each of which may have hundreds of racks of servers linked together through a massive switch with high-speed uplinks to handle all the traffic that the servers generate. That’s a traditional hierarchical design, which makes sense when most traffic goes on and off the Internet, said Najam Ahmad, vice president of network engineering.

The problem is, most of the communication in a Facebook data center now is just Facebook talking to itself. The applications that organize shared content, status updates and ads into the familiar news feed are highly distributed, so what the company calls “machine-to-machine” traffic is growing many times faster than the bits actually going out to the Internet.

Hundreds of racks per cluster meant hundreds of ports on the switch where all those racks link up. That’s an expensive and specialized need, and it was getting worse.

“We were already buying the largest box you can buy in the industry, and we were still hurting for more ports,” Ahmad said.

In addition, traffic between servers often has to get from one cluster to another, so the company had to constantly worry whether the links between those big clusters were fat enough.

What Facebook needed was a network that could keep carrying all those bits internally no matter how many there were or which servers they had to hit. So in place of those big clusters, it put together pods: much smaller groups of servers made up of just 48 racks.

Now Facebook just needs switches with 48 ports to link the racks in the pod and 48 more to connect with other switches that communicate with the rest of the pods. It’s much easier to buy those, and Facebook could even build them, Ahmad said.

With the new architecture, Facebook can supply 40-Gigabit ethernet pipes from any rack in the data center to any other. Rather than oversubscribing an uplink between two switches and assuming that all the racks won’t be sending data full-throttle all the time, it can equip the data center to handle maximum traffic all the time, a so-called non-blocking architecture.

The identical pods and standard fabric switches allow for easy expansion of both computing and network capacity, Facebook says.

“The architecture is such that you can continue to add pods until you run out of physical space or you run out of power,” Ahmad said. In Facebook’s case, the limiting factor is usually the amount of energy that’s available, he said.

The company has also developed software to automatically discover and configure new components and automate many management tasks. In fact, the fabric switches it uses have only standard, basic capabilities, with most other networking functions carried out by Facebook’s software.

Facebook has developed its own top-of-rack routing switch, called the Wedge, which wasn’t ready for the initial deployment in Altoona but can be used in the network fabric in the future. The architecture calls for servers to connect to the top of the rack over 10-Gigabit ethernet.

In Altoona, Facebook has been able to design the data center from the start for its new network architecture. It’s deployed fiber throughout the facility, installed core network equipment in the center and used straight-line fiber runs that are as short and direct as possible. When needed, the network fabric can be upgraded from 40-Gigabit ethernet to 100-Gigabit and beyond, Ahmad said.

In addition to sharing the core concepts of the network fabric, Facebook says it may later share designs and code with other companies. The company has made some of its data-center technologies available through the Open Compute Project that it founded in 2011.

Advertisements

About Five 9s Digital, LLC

Five 9s Digital is a data center real estate company offering data center development, investment sales, real estate advisory, and colocation and wholesale data center selection and procurement services. Five 9s Digital has been involved in the development and ownership of multiple mission critical facilities from single user build-to-suit facilities to multi-tenant data centers. Five 9s Digital brings a wealth of experience, background and expertise in providing solutions to the data center arena. Due to the specific and yet evolving requirements of mission critical end users in today’s world, Five 9s Digital is able to offer real solutions, real facilities, and real choices due its extensive and up-to-date knowledge base. Five 9s Digital is a Best Data Sites Certification Alliance™ member.
This entry was posted in Uncategorized and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s