A New Adaptive Gradient Method with Gradient Decomposition

Abstract

Adaptive gradient methods, especially Adam-type methods (such as Adam, AMSGrad, and AdaBound), have been proposed to speed up the training process with an element-wise scaling term on learning rates. However, they often generalize poorly compared with stochastic gradient descent (SGD) and its accelerated schemes such as SGD with momentum (SGDM). In this paper, we propose a new adaptive method called DecGD, which aims at achieving both good generalization like SGDM and rapid convergence like Adam-type methods. In particular, DecGD decomposes the current gradient into the product of two terms including a surrogate gradient and a loss vector. Our method adjusts the learning rates adaptively according to the current loss vector instead of the squared gradients used in Adam-type methods. The intuition for adaptive learning rates of DecGD is that a good optimizer needs to decrease the learning rates as the loss decreases, which is similar to the learning rates decay scheduling technique. Therefore, DecGD gets a rapid convergence by enabling learning rate decay in accordance with the loss vector. Convergence analysis is discussed in both convex and non-convex situations. Finally, empirical results on widely-used tasks demonstrate that DecGD shows better generalization performance than SGDM and offers the same rapid convergence as Adam-type methods.