The seamless integration of DOE's large-scale experimental facilities and powerful computing resources will help researchers keep pace with the ever-increasing influx of scientific data.

When the massive upgrade at the Advanced Photon Source (APS) at the U.S. Department of Energy's (DOE) Argonne National Laboratory is completed later this year, experiments at the powerful X-ray light source are expected to generate 100–200 petabytes, or 100-200 million gigabytes, of scientific data per year.

That's a substantial increase over the approximately five petabytes that were being produced annually at the APS, a DOE Office of Science user facility at Argonne, before the upgrade. And if you consider the DOE's four other light sources, the facilities are projected to yield an exabyte, or 1 billion gigabytes, of data per year in the coming decade.

"An exabyte of data is equivalent to streaming 1.5 million movies every day for a year," said Nicholas Schwarz, Argonne computer scientist and lead for scientific software and data management at the APS.

"But we need to do a lot more than simply move a lot of data around. For the X-ray experiments carried out at the APS, we need to use advanced computational tools to look at every pixel of every frame, analyze the data in near real time, and use the results to make decisions about the next experiment."

"To process all this data quickly, we require a lot of computing capabilities, from big computers and data storage, to analysis software, to the computational fabric that ties all of these resources together," he added.

The growing deluge of scientific data is not unique to light sources. Telescopes, particle accelerators, fusion research facilities, remote sensors and other scientific instruments also produce large amounts of data. And as their capabilities improve over time, the data generation rates will only continue to grow.

"The scientific community's ability to process, analyze, store and share these massive datasets is critical to gaining insights that will spark new discoveries," said Michael E. Papka, Argonne deputy associate laboratory director for Computing, Environment and Life Sciences.

Papka also serves as director of the Argonne Leadership Computing Facility (ALCF), a DOE Office of Science user facility at Argonne, and is a professor of computer science at the University of Illinois Chicago.

Argonne's Nexus effort is playing a pivotal role in advancing DOE's vision to build an integrated research infrastructure (IRI). Developing an IRI would accelerate data-intensive research by seamlessly integrating DOE's cutting-edge experimental facilities with its world-class supercomputing, artificial intelligence (AI) and data resources.

For over a decade, Argonne has been working to develop tools and methods to connect its powerful computing resources with large-scale experiments. Merging ALCF supercomputers with the APS has been a significant focus of the lab's IRI-related research, but the work has also included collaborations with the DIII-D National Fusion Facility in California and CERN's Large Hadron Collider in Switzerland. DIII-D is a DOE Office of Science user facility hosted by General Atomics.

More information: William Miller et al, Integrated Research Infrastructure Architecture Blueprint Activity (Final Report 2023). DOI: 10.2172/1984466