ESnet Turns on 400G Circuits to Four DOE National Labs, Supercharging Multi-Site Scientific Research

Today’s world-changing scientific research is being conducted by collaborators at far-flung national laboratories who require high-speed, low-latency access to high performance computing facilities and specialized instruments. The Energy Sciences Network (ESnet) is proud to announce that it has supercharged the current and future bandwidth for four of the Department of Energy’s (DOE’s) national laboratories and user facilities, unleashing 400 Gigabit per second (400G) capability for Argonne National Laboratory, National Energy Research Scientific Computing Center, Oak Ridge National Laboratory, and Pacific Northwest National Laboratory. With this boost in capacity, scientists can process, analyze, visualize, share, and store the enormous quantities of research data at speeds up to four times faster than previously possible. 

Rendering of ESnet6 showing the locations of the four sites with 400G capability in orange and the two in progress in yellow.

“It’s of vital importance that scientific researchers not be hindered by where they or their project’s instruments, computational resources, and data might be located,” said Inder Monga, executive director of ESnet. “Enabling 400G, which represents the networking industry’s current gold standard, will help facilitate that kind of seamless collaboration. We look forward to turning on 400G for more sites — and to upgrading to 800G as the technology begins to be available.” 

The 400G circuit installations were made possible by the 2022 launch of ESnet6, the sixth iteration of ESnet’s critical data circulatory system for the DOE Office of Science research complex. ESnet6 was specifically designed to support multi-facility collaborations aligned with the DOE’s new Integrated Research Infrastructure (IRI) initiative, to help DOE researchers and their international collaborators effectively harness the barrage of data generated by artificial intelligence, high-resolution instrument imagery, complex long-term global studies, and more. ESnet’s traffic is increasing by a factor of 10 every 5.5 years; in 2022, the total exceeded 1.36 exabytes. (An exabyte is equal to 1,000 petabytes or 1 billion gigabytes.)

What 400G Means for Big Science
 

Argonne has been involved in several collaborations that demonstrate the efficacy of integrating its supercomputers at the Argonne Leadership Computing Facility (ALCF) with experiments to accelerate scientific discoveries. Under the lab’s Nexus initiative, Argonne researchers are working with the DIII-D National Fusion Facility to enable on-demand access to ALCF supercomputers for experiment-time data analysis and predictive simulations that can be used to inform the parameters of the facility’s fast-paced plasma physics experiments. Similarly, Argonne researchers are demonstrating IRI capabilities through its ongoing efforts to tightly couple ALCF computing resources with experiments at Argonne’s Advanced Photon Source (APS), which is undergoing an upgrade that is expected to increase the volume of data generated by APS instruments by multiple orders of magnitude.

“In recent years, we’ve seen a surge in the near experiment-time analysis workflows being employed by the DOE light sources, fusion research facilities, and other large-scale experiments,” said ALCF Director Michael Papka. “As these facilities continue to evolve and improve, they’ll produce greater data volumes faster than ever before, increasing the demand for high-speed networking to the computing facilities. The ESnet upgrade is essential to keep pace with this growing scientific data deluge and meet future data-intensive research challenges.”

Oak Ridge National Laboratory (ORNL) leads the Earth Systems Grid Federation project (ESGF2) to improve the discovery, access, and storage of data used for Earth systems models and simulations and climate change research. The collaborative project, which includes Argonne and Lawrence Livermore National Laboratory, is already taking advantage of ESnet to move the planet’s largest collection of Earth System model output data between DOE high-performance computing and data facilities. The integration of ESnet is dramatically enhancing data-intensive science, facilitating data sharing, and improving access to simulation and AI platforms.

“The ESnet bandwidth upgrades at ORNL will simplify and accelerate data-integration intensive campaigns across DOE and collaborator facilities, further enabling and supporting integrated research infrastructures and projects like ESGF2,” said Mallikarjun Shankar, section head for Advanced Technologies at ORNL.   

Scientists at Pacific Northwest National Laboratory extract a foot-long soil core for study through the Molecular Observation Network. Just a single teaspoon of soil contains tens of thousands microbial species. Sorting out the proteins, metabolites, and interactions in the complex environment where soil, plants, and microbes meet results in a tremendous data stream. PNNL's new 400G connection to ESnet will enable even faster data sharing and analysis by collaborators around the nation. (Credit: Andrea Starr, Pacific Northwest National Laboratory)

And at the Environmental Molecular Sciences Laboratory, an Office of Science user facility located at PNNL, researchers have embarked on a project known as the Molecular Observation Network, a nationwide effort to understand the processes that govern what happens to carbon in soil. More carbon resides in Earth’s soil than in the atmosphere and vegetation combined, and its fate plays a huge role in our climate. It’s an incredibly active environment, with scientists collecting reams of data about soil processes. The upgrade makes it possible to transfer hundreds of gigabytes of project data in just a few minutes, not the hours previously required. The near real-time, constant data feed makes the autonomous experimentation planned at EMSL easier to implement.

“This ESNet6 upgrade presents a tremendously exciting opportunity for pursuing big science at DOE user facilities such as EMSL and at the other laboratories,” said Douglas Mans, EMSL director. “Accessing, moving, and storing massive data sets more effectively ensures U.S. scientific leadership and economic benefit.”

Speeding Up Data for High-Performance Computing

 
As one of ASCR’s three high performance computing facilities, NERSC will see an immediate benefit to the 400G upgrade. Increasingly, NERSC’s more than 9,000 users move data in and out of the data center in two main ways. The first is by transferring large datasets (simulated or from an experiment) between ASCR facilities as scientists take advantage of the specialized computing available at each site. The other is by moving data from DOE experimental and observational facilities for analysis. This analysis can be very time-sensitive, with scientists requiring quick turnarounds to inform running experiments.

“The ESnet6 upgrade is essential to maintaining scientific productivity as data is increasingly transferred between experiment and computing sites,” said Debbie Bard, group lead for Data Science Engagement at NERSC. “Enabling 400G is transformational – it opens up new ways of performing science integrated across multiple DOE facilities.”  

More 400G Capability on the Way

In the coming months, most of the data-intensive national labs will be upgrading to 400G service. CERN, the particle physics laboratory in Geneva whose Large Hadron Collider is one of ESnet’s largest data sources, and Lawrence Berkeley National Laboratory upgrades are almost complete. The greatly expanded capabilities offered by ESnet6 have positioned the DOE research ecosystem well to support the explosive growth in data generated by experimental observational and simulation science as well as AI/ML.

About ESnet

The Energy Sciences Network (ESnet) is a high-performance, unclassified network built to support and facilitate large-scale scientific research across America and around the world. Funded by the U.S. Department of Energy’s Office of Science and managed by Lawrence Berkeley National Laboratory, ESnet connects DOE’s national laboratory system, supercomputing facilities, and major scientific instruments, as well as peers with more than 270 research and commercial networks for global collaboration on the world’s biggest scientific challenges. ESnet’s vision is that scientific progress will be completely unconstrained by the physical location of instruments, people, computational resources, or data.