Michael Mahoney has led the Scientific Data Division’s AI and Learning Systems Group (AILearn) since 2022. Over the summer, he wrapped up his term as Group Lead and broadened his leadership to serve as the Scientific Data Division’s AI Initiative Research Lead. We caught up with Mahoney to reflect on what he’s done and what he’s planning to do next.

Can you describe what you’ve worked on as the AILearn Lead?

I would like to highlight my work on developing machine learning models that incorporate spatio-temporal and domain information in new ways and apply them to essential problems in seismology. This work, carried out with Ben Erichson and others in AILearn and with Rie Nakata and Nori Nakata in the Earth and Environmental Sciences Area, led to the discovery of novel machine learning models for predicting the ground motions for potential earthquakes. With one of our models, we predicted high-fidelity ground motions for future earthquakes for seismic hazard assessment and infrastructure resilience. With another, we demonstrated the capability to predict the intensity and timing of destructive ground motions rapidly. 

Even though we used seismological-specific data in this case, the machine learning models we developed are not domain-specific and can be used to solve a wide variety of scientific problems. We are currently exploring their potential in various other scientific contexts.

In addition to this work, AILearn is focused on several areas: applied mathematical questions at the foundation of scientific machine learning and randomized matrix algorithms; implementation questions about developing matrix algorithms and training neural network machine learning models at scale; and application-driven questions across multiple scientific domains.

Now that you’ve wrapped up your term as the AILearn Group Lead, what projects are you working on?

There are multiple projects, many of them connected to our SciGPT effort. SciGPT is a DOE ASCR-funded project that focuses on scaling questions for what are called scientific foundation models. Alongside this core work, we are exploring several related directions. 

I am very interested in discovering if we can apply the methodology of machine learning (basically, scale model and data and compute so no one of them is dominant) to a broad range of scientific data from a wide range of domains to develop something analogously broad and useful to text and image based foundation models (foundation models include large language models like OpenAI’s GPT and image models like DALL-E). 

What excites us is the potential for cross-domain adaptability. In many cases, models developed in one domain can be adapted to a different domain, and we are probing the limits of that transferability. For example, we are testing machine learning models built for subsurface modeling, such as water flow through complex media, and evaluating whether they can extend to cosmology, chemistry, nuclear physics, and beyond. Remarkably, evidence suggests that the embeddings from large language models (trained only on language data) can be fine-tuned with a relatively modest amount of domain data to produce high-quality time series forecasts in complex spatio-temporal applications that have nothing to do with text.

“The potential of data-driven methods to reshape how science is done is already becoming clear.”

What excites you the most about recent developments in AI?

The most interesting thing about recent developments in AI  and machine learning is how they shift the balance of power between data-driven methods and traditional scientific methods (by which I mean theory, computation, and experiment). The change is similar to the revolution brought about by the digital computer. Just as computers transformed science from being rooted mainly in theory and manual experiments into a process that integrated simulation, large-scale data analysis, and automated computation, AI is now opening the door to entirely new areas of scientific inquiry that were previously unimaginable.

The potential of data-driven methods to reshape how science is done is already becoming clear. While much attention is focused on how AI will change industries, and it will, the deeper impact may be on science itself, in ways as profound as the shift brought on by the widespread adoption of computers. The excitement comes from knowing we are beginning a transformation in how science is done.

Which papers should we read to learn more about your current work?

Recently, I have published several articles on new tools and approaches that I have helped to develop, including:

• XQuant — an algorithm that improves memory efficiency in machine learning models, outperforming many state-of-the-art methods. 

• Neural Discrete Equilibrium (NeurDE) — developed with Kareem Hegazy and collaborators at Rice University, this ML technique integrates physical conservation laws into machine learning to avoid common failure modes in hybrid modeling.

• FLEX — a backbone architecture for generative modeling of spatio-temporal physical systems using diffusion models.

• MatterChat – a versatile structure-aware multi-modal LLM that unifies material structural data and textual inputs into a single cohesive model. 

• Theoretical work on overloaded models — exploring what happens when machine learning models process too much information and how training methods can overcome these challenges.

Although certain aspects of this theoretical work are speculative, much of it offers practical insights for advancing scientific machine learning. Developing the applied mathematics and statistical methods underlying these approaches, in a way that conforms with both scientific computing and machine learning practices is necessary to deliver on the promise of scientific machine learning at scale.

To learn more about the other research happening in the Scientific Data Division, visit https://scidata.lbl.gov/