A-Z Index | Directory | Careers

Department of Energy’s ESnet Revving up to Unleash 100 Gbps National Science Network

October 24, 2012

Jon Bashor, Jbashor@lbl.gov, 510-486-5849


Most recent representation of ESnet's 100Gbps network

The Department of Energy’s Energy Sciences network, better known as ESnet, has long been at the forefront of providing reliable, high-bandwidth connectively to tens of thousands of researchers at national laboratories and universities across the country and to their colleagues around the world. Established in 1986, ESnet has continuously upgraded the speed, services and connectivity of the network. In late 2012, ESnet will take the next step, upgrading its network to 100 gigabits per second (Gbps), replacing the multiple 10 Gbps links now making up the network’s backbone.

Working with its partner Internet2, ESnet is currently building two network layers in parallel to prepare for the transition. The first network layer is the optical transport network, the bit-carrying foundation for the new 100 Gbps network known as ESnet5. The second layer is the routing and switching infrastructure being deployed to move data across the network. Those projects are major challenges in themselves, said ESnet Director Greg Bell, but the biggest hurdle will be the actual switchover of traffic to the new network.

Under the current schedule, the switchover to ESnet5 will be done in November 2012, with different regions being migrated in sequence. Determining the exact sequence to move traffic from an existing network infrastructure in full production mode to a totally new network build, without major outages to sites, is a combination of routing science and art that comes with many years of experience, according to Bell.

“It’s a difficult process involving dozens of steps,” Bell said. “The process is a bit like removing the running engine from one car as it speeds down the road and installing it in another car, also racing along the highway. But if we do it right, no one will notice the changeover.”

While the switch itself should not be noticeable, users may notice that the network will be able to immediately carry much more traffic in the form of massive datasets generated by scientists using DOE’s large-scale experimental facilities and supercomputing centers. According to Bell, the scientific data coming out of experiments at DOE’s particle accelerators, light sources and genome sequencing facilities are pushing the limits of ESnet’s current 10 Gbps connections.

“We never want the network to be the gating function for scientific discovery,” Bell said, “but we’re in danger of that happening. Several of our Labs could have benefited from this upgrade last year.”

In fact, ESnet did deploy a prototype 100 Gbps network last year connecting DOE supercomputing centers in California, Illinois and Tennessee, and a network peering point in New York. That project was the Advanced Networking Initiative funded under the Recovery Act. The successful prototype paved the way for the ESnet5 network.

The Advanced Networking Initiative represented a significant investment to help accelerate 100 Gbps technology and to help the scientific collaborations that fundamentally depend on high speed networks to get their science done, Bell said. Such collaborations can range from the international climate research community sharing datasets, to researchers from dozens of nations accessing results from experiments at the Large Hadron Collider, to small teams gathering images at light sources.

These massive datasets are significantly different than most Internet traffic, a situation Bell likens to elephants and mice. When looking at data flows, network engineers consider the size of the dataset being moved and the duration of the transport. Flows such as large scientific datasets, are called elephants, requiring more bandwidth and taking longer to move. Smaller flows, such as the routine messages, images and videos sent across the network, are much smaller and called mice. Keeping the two types of flows separate ensure that neither interferes with the other and both reach their destinations in one piece.

Because of its critical role in scientific discovery, ESnet is classified by the DOE as a national user facility. “ESnet5 is more than an infrastructure — a bigger, faster network. In reality, it’s a sophisticated instrument for data mobility,” Bell said.

To help both research institutions and industry explore new technologies for fully utilizing the 100 Gbps capability, ESnet also built a 100 Gbps testbed network as part of the Advanced Networking Initiative. About 25 different projects have been granted time on the dedicated testbed, some using technologies that could disrupt network traffic if tested on an open network. The testbed, accessible by reservation, is booked solid — up to 18 hours a day. Some of the work is at the leading edge, and teams are writing technical papers describing their research.

ESnet also provides users with tools and expertise to make the most of the network. OSCARS, ESnet's On-Demand Secure Circuits and Advance Reservation System, provides high-bandwidth virtual circuits that guarantee end-to-end data transfers across multiple domains. Originally a research concept, OSCARS has grown into a robust production service, and OSCARS virtual circuits currently carry 50 percent of ESnet’s annual 60 petabytes of traffic. A key aspect of OSCARS is its ability to create end-to-end circuits.

Delivering high bandwidth connectivity end to end is critical because data packets cross multiple network domains as they travel the Internet, and not all networks are configured the same. ESnet has helped develop perfSONAR, a tool for end-to-end monitoring and troubleshooting of multi-domain network performance. perfSONAR provides network engineers with the ability to test and measure network performance, as well as to archive data in order to pinpoint and solve service problems that may span multiple networks and international boundaries.

“Scientists don’t really care where the hiccup is, they just want to make sure their data gets to its destination,” said Inder Monga, ESnet’s Area Lead for Network Engineering and Research. “perfSONAR helps us identify the bottlenecks across domains, so we are able to keep the data flowing at the highest possible level of performance.”

Because network slowdowns can affect scientific discovery, ESnet staff also share their expertise with the larger community. The Fasterdata Knowledge Base is an open site providing a range of information for testing and tuning networks to speed up the flow of information.

About Computing Sciences at Berkeley Lab

High performance computing plays a critical role in scientific discovery. Researchers increasingly rely on advances in computer science, mathematics, computational science, data science, and large-scale computing and networking to increase our understanding of ourselves, our planet, and our universe. Berkeley Lab’s Computing Sciences Area researches, develops, and deploys new foundations, tools, and technologies to meet these needs and to advance research across a broad range of scientific disciplines.