A Technological Leap

In recent decades, people have become accustomed to what once felt like magic — the ability to stream video meetings, concerts, sports, and other events as they happen, to our computers and phones in high resolution, without jitter or lag. However, that technological leap, from broadcasting TV signals over the air to high-definition digital video over fiber-optic cables, took more than 50 years. And large-scale scientific research, involving data sets several orders of magnitude larger than a Netflix movie or YouTube livestream, still has hurdles to clear before achieving such ease of use.

It is much harder to stream science than science-fiction movies. The ability to execute what sounds like a simple task in 2024  –  turning on an instrument and watching the output as easily as Californians can watch a Florida rocket launch live  –  is a dance of far more complex bandwidth and performance needs. Right now, most large-scale instrument data must first be stored and then converted into files that can be processed and analyzed later.

The April 2024 test represented the culmination of nearly three years of collaboration between Jefferson Lab and ESnet to develop a novel networking hardware prototype that can connect scientific instruments to computing clusters over a wide-area network such as ESnet’s in real-time. The ESnet-JLab FPGA Accelerated Transport device, or EJFAT (pronounced “Edge-Fat”), uses traffic-shaping and load-balancing Field Programmable Gate Arrays that are designed to allow data from multiple types of scientific instruments to be streamed and processed in real-time (or near real-time) by multiple high-performance computing (HPC) facilities.

Such linking of the vast experimental and computational resources of the U.S. research enterprise is central to a broad initiative from the Department of Energy dubbed the Integrated Research Infrastructure (IRI). The DOE’s vision for IRI is “to empower researchers to meld DOE’s world-class research tools, infrastructure, and user facilities seamlessly and securely in novel ways to radically accelerate discovery and innovation.” (Learn more about IRI from the IRI Architecture Blueprint Activity Final Report.)

Contributing to that vision will be the HPDF project being developed by Jefferson Lab in partnership with Berkeley Lab. The first-of-its-kind resource for data science and research promises to transform the way scientific information is handled, transported, and stored among the many institutions across the DOE Office of Science’s six programs. Data streaming and processing could be key to realizing those ideas.

Connecting the Data Firehose

Stewarded by Berkeley Lab, ESnet is best known as the high-performance network that serves as the “data circulatory system” for the DOE’s tens of thousands of scientific researchers and their collaborators worldwide. In addition to providing advanced networking services, ESnet staff members work closely with scientific researchers and national laboratories’ networking experts to codesign new integrated approaches and solutions. In mid-2021, after learning they were both focused on solving the real-time streaming challenge, ESnet Planning & Innovation team members joined forces with Heyes and his team of scientific computing researchers and developers at Jefferson Lab.

At the time, Heyes was also serving as a subject matter expert for the IRI Architecture Blueprint Activity’s working group on the Time-Sensitive Pattern for science workflows. One of three common patterns identified by the developers of IRI (see ESnet’s meta-analysis report), the Time-Sensitive Pattern covers workflows that require real-time, or near-real-time, low-latency response across more than one facility or resource for timely decision-making and experiment control. Imagine stopping a telescope’s pan through the sky to stay focused on a supernova or quickly adjusting a nuclear fusion experiment’s “shots,” or short plasma discharges, to address plasma behaviors. Linking such unique research assets at multiple labs to function as one efficient “SuperLab” or “Superfacility” are concepts that have informed IRI.

Facilities with time-sensitive workflows include particle accelerators, X-ray light sources, and electron microscopes, all of which are fitted with numerous high-speed A/D data acquisition systems (DAQs) that can produce multiple 100- to 400-Gbps data streams for recording and processing. To date, this processing has been conducted on enormous banks of compute nodes in local and remote data facilities.

EJFAT is a tool for simplifying, automating, and accelerating how those raw data streams are quickly turned into actionable results. It could potentially work for all such instruments with time-sensitive workflows. ESnet and NERSC were awarded a Laboratory Directed Research and Development (LDRD) grant from the DOE to find a way to show the resilience of this approach.

A view from below the CLAS12 detector inside Experimental Hall B at Jefferson Lab.

For the April experiment, the EJFAT collaborators replayed data from Jefferson Lab’s Continuous Electron Beam Accelerator Facility (CEBAF) Large Acceptance Spectrometer (CLAS12). The device was not connected to the detector during a live experiment; instead, it replayed the streaming readout from a previous event. Direct connection of detectors to processing will be a requirement of the Electron-Ion Collider, a next-generation particle accelerator being built at Brookhaven National Laboratory in partnership with Jefferson Lab.

CEBAF’s array of detectors churns out tens of gigabytes of data per second. Newer instruments such as a fusion tokamak, X-ray synchrotron, or particle collider pump out terabytes of scientific data. (A terabyte is 1,000 gigabytes.) For comparison, Netflix estimates that streaming a high-definition movie requires 8 megabytes per second.

It’s hard to store all this data. At CEBAF, for example, researchers typically use a “trigger” system that records only interesting interactions. Triggered events are saved locally, and that subset is either processed locally or sent to DOE HPC centers such as NERSC, but traditional methods have not fully utilized ESnet’s high-speed infrastructure until now. Instead, the trigger system usually records events to files, and processing is limited by retrieval from storage and file-based batch transfers.

With data-stream in-network processing like EJFAT offers, raw data travels in one direction and processed data are streamed back in real time without the need for disk storage or buffering. This method requires no trigger and allows continuous data acquisition, filtering, and analysis. Rather than fetching water bucket by bucket, it’s more like turning on the bandwidth firehose and spraying at full power.

Debbie Bard, Department Head for Science Engagement and Workflows at NERSC, was part of the LDRD team that enabled EJFAT to connect to NERSC. “This is a real milestone, demonstrating a new capability that is sorely needed by many science teams who use NERSC,” Bard said. “We are increasingly seeing a need for automated, cross-facility workflows at DOE HPC facilities, which are often time-sensitive. HPC facilities all need to take occasional outages for maintenance and repairs. When this happens, science teams have to go through a very manual process of figuring out where they can send their data and reconfiguring their workflows to direct to a new site. EJFAT removes all this complexity, allowing scientists to process their data in a truly automated way across many sites, according to their capacity to accept work.”

There are other approaches to solving the streaming problem, most of which rely on software (versus EJFAT’s hardware-accelerated design) and require much more customization. They include Jefferson Lab’s own HOSS solution, which inspired EJFAT and uses RAM-based forward buffers to stream data from a particle physics pipeline. NERSC was also part of a successful project to stream data from X-ray scattering experiments using free-electron lasers (XFELs). The DELERIA project (for Distributed Event Level Experiment Readout and Analysis), another ESnet collaboration that is also part of Berkeley Lab’s LDRD program, seeks to generalize the data pipeline for the Gamma Ray Energy Tracking Array (GRETA) instrument by providing an interface between a streaming detector and data analysis running in a computing environment (see paper). These new approaches complement the file-based data transfer workflows managed by current distributed science projects, such as the University of Chicago’s nonprofit service Globus.

What’s Next