Quasi-Newton's method in the class gradient defined high-curvature subspace
Quasi-Newton's method in the class gradient defined high-curvature subspace
Classification problems using deep learning have been shown to have a high-curvature subspace in the loss landscape equal in dimension to the number of classes. Moreover, this subspace corresponds to the subspace spanned by the logit gradients for each class. An obvious strategy to speed up optimisation would be to use Newton's method in the high-curvature subspace and stochastic gradient descent in the co-space. We show that a naive implementation actually slows down convergence and we speculate why this might be.
Tuddenham, Mark
696df9f2-7a63-401e-9230-477c692f8782
Prugel-Bennett, Adam
b107a151-1751-4d8b-b8db-2c395ac4e14e
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
December 2020
Tuddenham, Mark
696df9f2-7a63-401e-9230-477c692f8782
Prugel-Bennett, Adam
b107a151-1751-4d8b-b8db-2c395ac4e14e
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Tuddenham, Mark, Prugel-Bennett, Adam and Hare, Jonathon
(2020)
Quasi-Newton's method in the class gradient defined high-curvature subspace.
12th Annual Workshop on Optimization for Machine Learning, Virtual.
11 Dec 2020.
Record type:
Conference or Workshop Item
(Paper)
Abstract
Classification problems using deep learning have been shown to have a high-curvature subspace in the loss landscape equal in dimension to the number of classes. Moreover, this subspace corresponds to the subspace spanned by the logit gradients for each class. An obvious strategy to speed up optimisation would be to use Newton's method in the high-curvature subspace and stochastic gradient descent in the co-space. We show that a naive implementation actually slows down convergence and we speculate why this might be.
This record has no associated files available for download.
More information
Published date: December 2020
Venue - Dates:
12th Annual Workshop on Optimization for Machine Learning, Virtual, 2020-12-11 - 2020-12-11
Identifiers
Local EPrints ID: 447201
URI: http://eprints.soton.ac.uk/id/eprint/447201
PURE UUID: f077abf4-b958-48e5-b7d3-7b61c72fa8bf
Catalogue record
Date deposited: 04 Mar 2021 17:46
Last modified: 17 Mar 2024 03:58
Export record
Contributors
Author:
Mark Tuddenham
Author:
Adam Prugel-Bennett
Author:
Jonathon Hare
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics