Advancing differentiable program optimisation via novel first and second-order metrics, and adaptive optimisation strategies

Abstract

This thesis is on the optimisation of deep neural networks, how they behave during training, how they can be made more efficient, and how methods and ideas from classical optimisation vary when applied in a deep learning context.

The first part of the thesis is a first-order analysis of the training process of deep neural networks, with a focus on the long range structure of the learning process and how well this can be approximated by a stochastic process.
We show that the learning process of deep neural networks can be reasonably well approximated by a stochastic process, and that this diffusion process can be used to understand the differences between the most popular optimisers including the artefacts from the update equations and the different convergence rates.
The second part of the thesis is on the second-order analysis of the training process of deep neural networks, with a focus on the curvature of the loss surface and how this can be used to improve the optimisation process.
In the third part we propose a new optimisation algorithm, called orthogonalised stochastic gradient descent (OSGD), which is based on the effect of introducing a diversification bias on the convolutional filters via orthonormalisation.

And also show that the adaption from SGD to OSGD can be used to improve the convergence rate of other optimisers, including Adam and RMSProp.

We show that this algorithm can be used to train deep neural networks with fewer epochs and better generalisation performance.

This work concludes with an overview of the results, a discussion of the implications of the work presented in this thesis, and some promising future directions of study.

More information

Identifiers

ORCID for Mark Tuddenham: orcid.org/0000-0002-3428-4051

ORCID for Jonathon Hare: orcid.org/0000-0003-2921-4283

Catalogue record

Export record