Demystifying parallel and distributed deep learning: An in-depth concurrency analysis
Deep Neural Networks (DNNs) are becoming an important tool in modern computing
applications. Accelerating their training is a major challenge and techniques range from …
applications. Accelerating their training is a major challenge and techniques range from …
Collective motion
We review the observations and the basic laws describing the essential aspects of collective
motion—being one of the most common and spectacular manifestation of coordinated …
motion—being one of the most common and spectacular manifestation of coordinated …
Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent
Most distributed machine learning systems nowadays, including TensorFlow and CNTK, are
built in a centralized fashion. One bottleneck of centralized algorithms lies on high …
built in a centralized fashion. One bottleneck of centralized algorithms lies on high …
Consensus and cooperation in networked multi-agent systems
R Olfati-Saber, JA Fax, RM Murray - Proceedings of the IEEE, 2007 - ieeexplore.ieee.org
This paper provides a theoretical framework for analysis of consensus algorithms for multi-
agent networked systems with an emphasis on the role of directed information flow …
agent networked systems with an emphasis on the role of directed information flow …
Distributed subgradient methods for multi-agent optimization
We study a distributed computation model for optimizing a sum of convex objective functions
corresponding to multiple agents. For solving this (not necessarily smooth) optimization …
corresponding to multiple agents. For solving this (not necessarily smooth) optimization …
Randomized gossip algorithms
Motivated by applications to sensor, peer-to-peer, and ad hoc networks, we study distributed
algorithms, also known as gossip algorithms, for exchanging information and for computing …
algorithms, also known as gossip algorithms, for exchanging information and for computing …
Network topology and communication-computation tradeoffs in decentralized optimization
In decentralized optimization, nodes cooperate to minimize an overall objective function that
is the sum (or average) of per-node private objective functions. Algorithms interleave local …
is the sum (or average) of per-node private objective functions. Algorithms interleave local …
Constrained consensus and optimization in multi-agent networks
We present distributed algorithms that can be used by multiple agents to align their
estimates with a particular value over a network with time-varying connectivity. Our …
estimates with a particular value over a network with time-varying connectivity. Our …
Decentralized federated averaging
Federated averaging (FedAvg) is a communication-efficient algorithm for distributed training
with an enormous number of clients. In FedAvg, clients keep their data locally for privacy …
with an enormous number of clients. In FedAvg, clients keep their data locally for privacy …
Asynchronous decentralized parallel stochastic gradient descent
Most commonly used distributed machine learning systems are either synchronous or
centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a …
centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a …