The MFIX-Exa software subproject of the U.S. Department of Energy’s (DOE’s) Exascale Computing Project (ECP) is helping achieve that scaling. Utilizing the AMReX software framework developed at Lawrence Berkeley National Laboratory (Berkeley Lab), MFIX-Exa can track billions of individual particles simultaneously to simulate the gas-solid flows typical of a reactor – something standard computing cannot achieve.
The direct scale-up of multiphase flow reactors is known to be unreliable, and the current approach requires building and testing physical systems, which is cost prohibitive. MFIX-Exa will leverage exascale computing power to model emerging CCS technologies to aid in the design and optimization of these systems, helping control costs and reducing the risk of failure. MFIX-Exa can recognize when things go wrong far more rapidly than in a lab or experimental setting.
MFIX-Exa is a joint project between DOE’s National Energy Technology Laboratory (NETL) and Berkeley Lab, powered by a multi-lab team that integrates expertise in HPC directly with expertise in multiphase flow modeling. Jordan Musser, a scientist at NETL, is the MFIX-Exa principal investigator, while Berkeley Lab scientists Weiqun Zhang and Andrew Myers are the lead AMReX developers. Hengjie Wang, a postdoc in Berkeley Lab’s Center for Computational Sciences and Engineering (CCSE), contributes to MFIX-Exa as well, working on balancing the computation load from modeling different regions of the chemical reactors, which has led to significant computing speed improvements
“We had a good amount of experience using AMReX with particle-in-cell codes,” said Myers, a computer systems engineer in CCSE and a member of the AMReX core development team. “But MFIX-Exa was the first AMReX-based code to have direct pairwise particle interactions, so there was a set of tools we needed to build out to support those operations; because these particles are directly interacting with one another, it gets much more computationally expensive.”
Using AMReX allowed the MFIX-Exa project to easily migrate from running on only CPUs to running on hybrid, GPU-accelerated architectures. “When the MFIX-Exa project started, we started with MFIX legacy code, but it could only run on CPUs,” said CCSE senior engineer Weiqun Zhang. “But in HPC the hardware is changing and the next-generation supercomputing uses GPUs. So AMReX played an important role because the code can be used on CPUs or various types of GPUs.”
The code is geared toward simulating commercial chemical looping reactors that can be used to reduce CO2 emissions through capture and storage. The MFIX-Exa code calculates the fluid and particle dynamics in these types of reactors and can simulate the behavior of large-scale commercial reactors during their design phase. This allows engineers to prototype reactors with a high-fidelity model and diagnose issues before a plant is built. MFIX-Exa will allow a ramp-up of problem size, speed, and accuracy on exascale computers over the next decade.
MFIX-Exa is one of six ECP codes built on AMReX; other AMReX-based applications in ECP include accelerator modeling, cosmology and astrophysics, combustion and wind energy modeling. The MFIX-Exa code itself has been used to launch several offshoot codes, which have actually spawned some new research directions, according to Berkeley Lab Applied Mathematics Department Head Ann Almgren.
“AMReX has always had a very symbiotic relationship with the application codes built using AMReX; the application codes benefit immensely from being able to take advantage of the common infrastructure,” she said. “And when one application needs a new feature that AMReX can supply, that feature is immediately available to all the other applications. This synergy has enabled us to go in new directions we never foresaw.”
This article incorporates information from an ECP technology highlight about the MFIX-Exa project.
About Computing Sciences at Berkeley Lab
High performance computing plays a critical role in scientific discovery. Researchers increasingly rely on advances in computer science, mathematics, computational science, data science, and large-scale computing and networking to increase our understanding of ourselves, our planet, and our universe. Berkeley Lab’s Computing Sciences Area researches, develops, and deploys new foundations, tools, and technologies to meet these needs and to advance research across a broad range of scientific disciplines.
About Computing Sciences at Berkeley Lab
High performance computing plays a critical role in scientific discovery. Researchers increasingly rely on advances in computer science, mathematics, computational science, data science, and large-scale computing and networking to increase our understanding of ourselves, our planet, and our universe. Berkeley Lab's Computing Sciences Area researches, develops, and deploys new foundations, tools, and technologies to meet these needs and to advance research across a broad range of scientific disciplines.