Publication Type

Journal Article

Version

acceptedVersion

Publication Date

1-2025

Abstract

We revisit two fundamental decentralized optimization methods, Decentralized Gradient Tracking (DGT) and Decentralized Gradient Descent (DGD), with multiple local updates. We consider two settings and demonstrate that incorporating local update steps can reduce communication complexity. Specifically, for  $\mu$-strongly convex and $L$-smooth loss functions, we proved that local DGT  achieves communication complexity {}{$\tilde{\mathcal{O}} \Big(\frac{L}{\mu(K+1)} + \frac{\delta + {}{\mu}}{\mu (1 - \rho)} + \frac{\rho }{(1 - \rho)^2} \cdot \frac{L+ \delta}{\mu}\Big)$}, where $K$ is the number of additional local update}, $\rho$ measures the network connectivity and $\delta$ measures the second-order heterogeneity of the local losses. Our results reveal the tradeoff between communication and computation and show increasing $K$ can effectively reduce communication costs when the data heterogeneity is low and the network is well-connected. We then consider the over-parameterization regime where the local losses share the same minimums. We proved that employing local updates in DGD, even without gradient correction, achieves exact linear convergence under the Polyak-Łojasiewicz (PL) condition, which can yield a similar effect as DGT in reducing communication complexity. Customization of the result to linear models is further provided, with improved rate expression. Numerical experiments validate our theoretical results.

Discipline

Artificial Intelligence and Robotics

Research Areas

Data Science and Engineering; Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

IEEE Transactions on Signal Processing

Volume

73

First Page

751

Last Page

765

ISSN

1053-587X

Identifier

10.1109/TSP.2025.3533208

Publisher

Institute of Electrical and Electronics Engineers

Additional URL

https://doi.org/10.1109/TSP.2025.3533208

Download

Share

COinS