In many domains, the previous decade was characterized by increasing data volumes and
growing complexity of data analyses, creating new demands for batch processing on …

Science and engineering applications are now generating data at an unprecedented rate.
From large facilities such as the Large Hadron Collider to portable DNA sequencing …

Extremely high data rates at modern synchrotron and X-ray free-electron laser light source
beamlines motivate the use of machine learning methods for data reduction, feature …

Extreme-scale simulations and experiments can generate large amounts of data, whose
volume can exceed the compute and/or storage capacity at the simulation or experimental …

Wide area data transfers play an important role in many science applications but rely on
expensive infrastructure that often delivers disappointing performance in practice. In …

High Performance Computing (HPC) is an important method for scientific discovery via large-
scale simulation, data analysis, or artificial intelligence. Leadership-class supercomputers …

Wide area file transfers play an important role in many science applications. File transfer
tools typically deliver the highest performance for datasets with a small number of large files …

Modern scientific instruments, such as detectors at synchrotron light sources, generate data
at such high rates that online processing is needed for data reduction, feature detection …

Many extreme-scale applications require the movement of large quantities of data to, from,
and among leadership computing facilities, as well as other scientific facilities and the home …

We describe best practices for providing convenient, high-speed, secure access to large
data via research data portals. We capture these best practices in a new design pattern, the …