The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication (Extended Abstract)

The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication (Extended Abstract)

Blake Woodworth, Brian Bullins, Ohad Shamir, Nathan Srebro

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
Sister Conferences Best Papers. Pages 5359-5363. https://doi.org/10.24963/ijcai.2022/751

We resolve the min-max complexity of distributed stochastic convex optimization (up to a log factor) in the intermittent communication setting, where M machines work in parallel over the course of R rounds of communication to optimize the objective, and during each round of communication, each machine may sequentially compute K stochastic gradient estimates. We present a novel lower bound with a matching upper bound that establishes an optimal algorithm.
Keywords:
Artificial Intelligence: General