The University of Southampton
University of Southampton Institutional Repository

Quasi-Newton's method in the class gradient defined high-curvature subspace

Quasi-Newton's method in the class gradient defined high-curvature subspace
Quasi-Newton's method in the class gradient defined high-curvature subspace
Classification problems using deep learning have been shown to have a high-curvature subspace in the loss landscape equal in dimension to the number of classes. Moreover, this subspace corresponds to the subspace spanned by the logit gradients for each class. An obvious strategy to speed up optimisation would be to use Newton's method in the high-curvature subspace and stochastic gradient descent in the co-space. We show that a naive implementation actually slows down convergence and we speculate why this might be.
Tuddenham, Mark
696df9f2-7a63-401e-9230-477c692f8782
Prugel-Bennett, Adam
b107a151-1751-4d8b-b8db-2c395ac4e14e
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Tuddenham, Mark
696df9f2-7a63-401e-9230-477c692f8782
Prugel-Bennett, Adam
b107a151-1751-4d8b-b8db-2c395ac4e14e
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9

Tuddenham, Mark, Prugel-Bennett, Adam and Hare, Jonathon (2020) Quasi-Newton's method in the class gradient defined high-curvature subspace. 12th Annual Workshop on Optimization for Machine Learning, Virtual. 11 Dec 2020.

Record type: Conference or Workshop Item (Paper)

Abstract

Classification problems using deep learning have been shown to have a high-curvature subspace in the loss landscape equal in dimension to the number of classes. Moreover, this subspace corresponds to the subspace spanned by the logit gradients for each class. An obvious strategy to speed up optimisation would be to use Newton's method in the high-curvature subspace and stochastic gradient descent in the co-space. We show that a naive implementation actually slows down convergence and we speculate why this might be.

Full text not available from this repository.

More information

Published date: December 2020
Venue - Dates: 12th Annual Workshop on Optimization for Machine Learning, Virtual, 2020-12-11 - 2020-12-11

Identifiers

Local EPrints ID: 447201
URI: http://eprints.soton.ac.uk/id/eprint/447201
PURE UUID: f077abf4-b958-48e5-b7d3-7b61c72fa8bf
ORCID for Mark Tuddenham: ORCID iD orcid.org/0000-0002-3428-4051
ORCID for Jonathon Hare: ORCID iD orcid.org/0000-0003-2921-4283

Catalogue record

Date deposited: 04 Mar 2021 17:46
Last modified: 05 Mar 2021 02:57

Export record

Contributors

Author: Mark Tuddenham ORCID iD
Author: Adam Prugel-Bennett
Author: Jonathon Hare ORCID iD

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×