The University of Southampton
University of Southampton Institutional Repository

A variance controlled stochastic method with biased estimation for faster non-convex optimization

A variance controlled stochastic method with biased estimation for faster non-convex optimization
A variance controlled stochastic method with biased estimation for faster non-convex optimization
This paper proposes a new novelty optimization method Variance Controlled Stochastic Gradient (VCSG) to improve the performance of the stochastic variance reduced gradient (SVRG) algorithm. To avoid over-reducing the variance of gradient by SVRG, a hyper-parameter λ is introduced in VCSG that is able to control the reduced variance of SVRG. Theory shows that the optimization method can converge by using an unbiased gradient estimator, but in practice, biased gradient estimation can allow more efficient convergence to the vicinity since an unbiased approach is computationally more expensive. λ also has the effect of balancing the trade-off between unbiased and biased estimations. Secondly, to minimize the number of full gradient calculations in SVRG, a variance-bounded batch is introduced to reduce the number of gradient calculations required in each iteration. For smooth non-convex functions, the proposed algorithm converges to an approximate first-order stationary point (i.e. Ek∇f(x)k 2 ≤ ) within O(min{1/ 3/2 , n1/4 / }) number of stochastic gradient evaluations, which improves the leading gradient complexity of stochastic gradient-based method SCSG (O(min{1/ 5/3 , n2/3 / }) [20]. It is shown theoretically and experimentally that VCSG can be deployed to improve convergence.
Computational complexity, Deep learning, Non-convex optimization
0302-9743
135 - 150
Bi, Jia
e07a78d1-62dd-4b1d-b223-4107aa3627c7
Gunn, Stephen R.
306af9b3-a7fa-4381-baf9-5d6a6ec89868
Bi, Jia
e07a78d1-62dd-4b1d-b223-4107aa3627c7
Gunn, Stephen R.
306af9b3-a7fa-4381-baf9-5d6a6ec89868

Bi, Jia and Gunn, Stephen R. (2021) A variance controlled stochastic method with biased estimation for faster non-convex optimization. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 135 - 150. (doi:10.1007/978-3-030-86523-8_9).

Record type: Article

Abstract

This paper proposes a new novelty optimization method Variance Controlled Stochastic Gradient (VCSG) to improve the performance of the stochastic variance reduced gradient (SVRG) algorithm. To avoid over-reducing the variance of gradient by SVRG, a hyper-parameter λ is introduced in VCSG that is able to control the reduced variance of SVRG. Theory shows that the optimization method can converge by using an unbiased gradient estimator, but in practice, biased gradient estimation can allow more efficient convergence to the vicinity since an unbiased approach is computationally more expensive. λ also has the effect of balancing the trade-off between unbiased and biased estimations. Secondly, to minimize the number of full gradient calculations in SVRG, a variance-bounded batch is introduced to reduce the number of gradient calculations required in each iteration. For smooth non-convex functions, the proposed algorithm converges to an approximate first-order stationary point (i.e. Ek∇f(x)k 2 ≤ ) within O(min{1/ 3/2 , n1/4 / }) number of stochastic gradient evaluations, which improves the leading gradient complexity of stochastic gradient-based method SCSG (O(min{1/ 5/3 , n2/3 / }) [20]. It is shown theoretically and experimentally that VCSG can be deployed to improve convergence.

Text
Springer_Lecture_Notes_in_Computer_Science__1_ - Accepted Manuscript
Download (507kB)

More information

e-pub ahead of print date: 11 September 2021
Keywords: Computational complexity, Deep learning, Non-convex optimization

Identifiers

Local EPrints ID: 452393
URI: http://eprints.soton.ac.uk/id/eprint/452393
ISSN: 0302-9743
PURE UUID: b788c0c4-a3e7-4501-a447-6c23453a2e8e

Catalogue record

Date deposited: 09 Dec 2021 17:55
Last modified: 27 Apr 2022 04:53

Export record

Altmetrics

Contributors

Author: Jia Bi
Author: Stephen R. Gunn

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×