Stochastic relaxed inertial forward-backward-forward splitting for monotone inclusions in Hilbert spaces
Stochastic relaxed inertial forward-backward-forward splitting for monotone inclusions in Hilbert spaces
We consider monotone inclusions defined on a Hilbert space where the operator is given by the sum of a maximal monotone operator T and a single-valued monotone, Lipschitz continuous, and expectation-valued operator V. We draw motivation from the seminal work by Attouch and Cabot (Attouch in AMO 80:547–598, 2019, Attouch in MP 184: 243–287) on relaxed inertial methods for monotone inclusions and present a stochastic extension of the relaxed inertial forward–backward-forward method. Facilitated by an online variance reduction strategy via a mini-batch approach, we show that our method produces a sequence that weakly converges to the solution set. Moreover, it is possible to estimate the rate at which the discrete velocity of the stochastic process vanishes. Under strong monotonicity, we demonstrate strong convergence, and give a detailed assessment of the iteration and oracle complexity of the scheme. When the mini-batch is raised at a geometric (polynomial) rate, the rate statement can be strengthened to a linear (suitable polynomial) rate while the oracle complexity of computing an ϵ-solution improves to O(1/ϵ). Importantly, the latter claim allows for possibly biased oracles, a key theoretical advancement allowing for far broader applicability. By defining a restricted gap function based on the Fitzpatrick function, we prove that the expected gap of an averaged sequence diminishes at a sublinear rate of O(1/k) while the oracle complexity of computing a suitably defined ϵ-solution is O(1/ϵ1+a) where a>1. Numerical results on two-stage games and an overlapping group Lasso problem illustrate the advantages of our method compared to competitors.
Complexity, Dynamic sampling, Monotone operator splitting, Stochastic approximation, Variance reduction
465-524
Cui, Shisheng
56dd0856-dc0f-40ad-8c55-9fdbac4aa1e8
Shanbhag, Uday
9d7babef-cf38-4d51-a969-447a188ba0d5
Staudigl, Mathias
d5078a05-5eaf-48ee-9789-360d4f27902d
Vuong, Phan Tu
52577e5d-ebe9-4a43-b5e7-68aa06cfdcaf
November 2022
Cui, Shisheng
56dd0856-dc0f-40ad-8c55-9fdbac4aa1e8
Shanbhag, Uday
9d7babef-cf38-4d51-a969-447a188ba0d5
Staudigl, Mathias
d5078a05-5eaf-48ee-9789-360d4f27902d
Vuong, Phan Tu
52577e5d-ebe9-4a43-b5e7-68aa06cfdcaf
Cui, Shisheng, Shanbhag, Uday, Staudigl, Mathias and Vuong, Phan Tu
(2022)
Stochastic relaxed inertial forward-backward-forward splitting for monotone inclusions in Hilbert spaces.
Computational Optimization and Applications, 83 (2), .
(doi:10.1007/s10589-022-00399-3).
Abstract
We consider monotone inclusions defined on a Hilbert space where the operator is given by the sum of a maximal monotone operator T and a single-valued monotone, Lipschitz continuous, and expectation-valued operator V. We draw motivation from the seminal work by Attouch and Cabot (Attouch in AMO 80:547–598, 2019, Attouch in MP 184: 243–287) on relaxed inertial methods for monotone inclusions and present a stochastic extension of the relaxed inertial forward–backward-forward method. Facilitated by an online variance reduction strategy via a mini-batch approach, we show that our method produces a sequence that weakly converges to the solution set. Moreover, it is possible to estimate the rate at which the discrete velocity of the stochastic process vanishes. Under strong monotonicity, we demonstrate strong convergence, and give a detailed assessment of the iteration and oracle complexity of the scheme. When the mini-batch is raised at a geometric (polynomial) rate, the rate statement can be strengthened to a linear (suitable polynomial) rate while the oracle complexity of computing an ϵ-solution improves to O(1/ϵ). Importantly, the latter claim allows for possibly biased oracles, a key theoretical advancement allowing for far broader applicability. By defining a restricted gap function based on the Fitzpatrick function, we prove that the expected gap of an averaged sequence diminishes at a sublinear rate of O(1/k) while the oracle complexity of computing a suitably defined ϵ-solution is O(1/ϵ1+a) where a>1. Numerical results on two-stage games and an overlapping group Lasso problem illustrate the advantages of our method compared to competitors.
Text
s10589-022-00399-3
- Version of Record
More information
Accepted/In Press date: 30 June 2022
e-pub ahead of print date: 30 July 2022
Published date: November 2022
Additional Information:
Funding Information:
M. Staudigl acknowledges support from the COST Action CA16228 “European Network for Game Theory”.
Publisher Copyright:
© 2022, The Author(s).
Keywords:
Complexity, Dynamic sampling, Monotone operator splitting, Stochastic approximation, Variance reduction
Identifiers
Local EPrints ID: 469068
URI: http://eprints.soton.ac.uk/id/eprint/469068
ISSN: 0926-6003
PURE UUID: 760655ea-dd2e-46cb-b64a-1128d4881ce5
Catalogue record
Date deposited: 06 Sep 2022 17:57
Last modified: 17 Mar 2024 03:58
Export record
Altmetrics
Contributors
Author:
Shisheng Cui
Author:
Uday Shanbhag
Author:
Mathias Staudigl
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
