We propose TopoOpt, a novel direct-connect fabric for deep neural network (DNN) training
workloads. TopoOpt co-optimizes the distributed training process across three dimensions …

Recent trends in high-performance computing and deep learning have led to the
proliferation of studies on large-scale deep neural network training. However, the frequent …

Deep learning's widespread adoption in various fields has made distributed training across
multiple computing nodes essential. However, frequent communication between nodes can …

In large-scale distributed machine learning (DML), the synchronization efficiency of the
distributed algorithm becomes a critical factor that affects the training time of machine …

Gradient compression is a promising approach to alleviating the communication bottleneck
in data parallel deep neural network (DNN) training by significantly reducing the data …

The distributed alternating direction method of multipliers (ADMM) is an effective algorithm
for solving large-scale optimization problems. However, its high communication cost limits its …

In this paper, we present COFFEE, cross-layer optimization for fast and efficient executions
of the Sinkhorn-Knopp (SK) algorithm on HPC systems with clusters of compute nodes by …

In the last decade, DL training has emerged as an HPC-scale workload running on large
clusters, the size of the largest supercomputers on the Top500 list. The dominant …

We are exploring a new concept of error permissive computing that improves the capability
and capacity while drastically reducing power consumption. More specifically, we …