The University of Southampton
University of Southampton Institutional Repository

Global and quadratic convergence of Newton hard-thresholding pursuit

Global and quadratic convergence of Newton hard-thresholding pursuit
Global and quadratic convergence of Newton hard-thresholding pursuit

Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. This paper provides a theoretical justification for the use of the restricted Newton step. We build our theory and algorithm, Newton Hard-Thresholding Pursuit (NHTP), for the sparsity-constrained optimization. Our main result shows that NHTP is quadratically convergent under the standard assumption of restricted strong convexity and smoothness. We also establish its global convergence to a stationary point under a weaker assumption. In the special case of the compressive sensing, NHTP e

Global convergence, Hard thresholding, Newton's method, Quadratic convergence rate, Sparse optimization, Stationary point
1-45
Zhou, Shenglong
d183edc9-a9f6-4b07-a140-a82213dbd8c3
Xiu, Naihua
8b5770f7-ae35-4dbe-884a-02fb4ea27bee
Qi, Hou-Duo
e9789eb9-c2bc-4b63-9acb-c7e753cc9a85
Zhou, Shenglong
d183edc9-a9f6-4b07-a140-a82213dbd8c3
Xiu, Naihua
8b5770f7-ae35-4dbe-884a-02fb4ea27bee
Qi, Hou-Duo
e9789eb9-c2bc-4b63-9acb-c7e753cc9a85

Zhou, Shenglong, Xiu, Naihua and Qi, Hou-Duo (2021) Global and quadratic convergence of Newton hard-thresholding pursuit. Journal of Machine Learning Research, 22, 1-45, [12].

Record type: Article

Abstract

Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. This paper provides a theoretical justification for the use of the restricted Newton step. We build our theory and algorithm, Newton Hard-Thresholding Pursuit (NHTP), for the sparsity-constrained optimization. Our main result shows that NHTP is quadratically convergent under the standard assumption of restricted strong convexity and smoothness. We also establish its global convergence to a stationary point under a weaker assumption. In the special case of the compressive sensing, NHTP e

Text
Global - Accepted Manuscript
Available under License Creative Commons Attribution.
Download (1MB)

More information

Submitted date: 9 January 2019
Accepted/In Press date: 3 February 2021
Published date: 11 February 2021
Additional Information: Funding Information: We would like to acknowledge support for this project from the National Natural Science Foundation of China (11971052,12011530155), “111” Project of China (B16002), The Alan Turing Institute and the Royal Society International Exchange Programme (IEC-NSFC191543). We particularly thank the referee who went through our technical proofs and offered us valuable suggestions on the condition (14). We also thank Prof Ziyan Luo of Beijing Jiaotong University for helping to improve the proof of Thm. 10. Publisher Copyright: © 2021 Microtome Publishing. All rights reserved.
Keywords: Global convergence, Hard thresholding, Newton's method, Quadratic convergence rate, Sparse optimization, Stationary point

Identifiers

Local EPrints ID: 433233
URI: http://eprints.soton.ac.uk/id/eprint/433233
PURE UUID: 2c56acb9-cf90-46bb-9759-b6b7e253d9f1
ORCID for Shenglong Zhou: ORCID iD orcid.org/0000-0003-2843-1614
ORCID for Hou-Duo Qi: ORCID iD orcid.org/0000-0003-3481-4814

Catalogue record

Date deposited: 12 Aug 2019 16:30
Last modified: 29 Oct 2022 01:40

Export record

Contributors

Author: Shenglong Zhou ORCID iD
Author: Naihua Xiu
Author: Hou-Duo Qi ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×