Workflow and Data Management

Experimental, observational, and theory-based scientific data presents complex challenges in quality assurance, quality control, data integration, data reduction, analysis and visualization, real-time decision-making, human-in-the loop, monitoring, and steering. To address these challenges, we create integrated systems to manage the data coming from instruments and computations. These systems perform streaming data analytics over high-speed wide-area networks such as the Energy Sciences Network (ESnet), processing data in real time at high-performance computing (HPC) facilities such as the National Energy Research Scientific Computing Center (NERSC). We employ statistical methods and/or surrogate models to process the data concurrently and close the loop to communicate back to the data source. To store and archive the data for offline analysis, we create intelligent and automated data- and resource-management architectures.

Projects

The Superfacility Project

The Superfacility concept is a framework for integrating experimental and observational instruments with computational and data facilities. Data produced by light sources, microscopes, telescopes, and other devices can stream in real time to large computing facilities where it can be analyzed, archived, curated, combined with simulation data, and served to the science user community via powerful computing, storage, and networking systems. The NERSC Superfacility Project is designed to identify the technical and policy challenges in this concept for an HPC center. Contact: Debbie Bard

Toward Self-Guiding Field Laboratories to Support Water Management

In Earth and environmental sciences, applications such as resilient infrastructure, predictions of ecosystem responses to climate change and disturbance, and monitoring of energy resources need to collect heterogeneous data from field environments and infer scientific insights or make decisions based on predictions generated by models. This project develops a solid foundation for self-guiding field laboratories (SGFL) to facilitate adaptive measurements driven in near-real-time by synthesizing lab and field observations with models. Contact: Yuxin Wu 

News