A Comprehensive Look at High Performance Parallel I/O

In the 1990s, high performance computing (HPC) made a dramatic transition to massively parallel processors. As this model solidified over the next 20 years, supercomputing performance increased from gigaflops—billions of calculations per second—to petaflops—quadrillions of calculations per second—generating a tsunami of data along the way.

In this era of “big data,” high performance parallel I/O—the way disk drives efficiently read and write information on HPC systems—is extremely important. Yet, the last book to summarize best practices in this area was written more than 10 years ago. To fill the void, Prabhat of the Lawrence Berkeley National Laboratory (Berkeley Lab) and Quincey Koziol of the HDF Group brought together leading practitioners, researchers, software architects, developers and scientists to contribute their insights for a new book called “High Performance Parallel I/O.”

Published on October 24, the editors will be signing copies of the book and answering questions in the Department of Energy’s booth (#1939) at SC14 (Supercomputing Conference, 2014) on Tuesday, November 18 at 5:15-6:00 p.m.

“This book provides a survey of the significant accomplishments in file systems, libraries and tools that have been developed for about a decade. These developments have now reached a state of relative maturity and are ripe for a treatment in book format,” writes Horst Simon, Deputy Director of Berkeley Lab in the book’s forward. “I highly recommend this timely book for computational scientists and engineers.”

According to Prabhat, the book is divided into six parts. The first part explains how large-scale HPC facilities scope, configure, and operate systems, with an emphasis on choices of I/O hardware, middleware, and applications. It then traverses up the I/O software stack. The second part covers the file system layer and the third part discusses middleware and user-facing libraries.

Delving into real-world scientific applications that use the parallel I/O infrastructure, he notes that the fourth part presents case studies from particle-in-cell, stochastic, finite volume, and direct numerical simulations. The fifth part gives an overview of various profiling and benchmarking tools used by practitioners. The final part, addresses the implications of current trends in HPC on parallel I/O in the exascale world.

“This book is a soup to nuts collection of everything you need to know about high performance parallel I/O, from hardware all the way up into the software layers and future work,” says Koziol.

For more information about the book: http://www.crcpress.com/product/isbn/9781466582347

About the Editors:

Prabhat leads the NERSC Data and Analytics Services Group at Berkeley Lab. His main research interests include Big Data analytics, scientific data management, parallel I/O, HPC, and scientific visualization. He is also interested in atmospheric science and climate change. 

Quincey Koziol is the director of core software development and HPC at The HDF Group, where he leads the HDF5 software project. His research interests include HPC, scientific data storage, and software engineering and management.