The University of Southampton
University of Southampton Institutional Repository

Deep cascade learning

Deep cascade learning
Deep cascade learning
Deep Learning has demonstrated outstanding performance on several machine learning tasks. These results are attributed to training very deep networks on large scale datasets. In this thesis we investigate training models in a layer-wise fashion. We quantify performance and discuss the advantages of using such training algorithms on computer vision and signal processing tasks.

Inspired by the Cascade Correlation algorithm, which is a growing neural network that iteratively learns artificial neurons, we developed a supervised layer-wise training algorithm, which we name Deep Cascade Learning. Our methodology takes as input the architecture to train and splits the model in submodels, where each iteration trains only one layer of the network. The feature representation gets more robust as layers are stacked. Moreover, the algorithm provides training complexity reduction while preserving competitive results in comparison with state-of-the-art end to end training. We demonstrate these advantages on multiple benchmark datasets.

Given that Deep Cascade Learning trains models from scratch successfully, we also look at layer-wise methods to transfer features from a large base dataset, to a smaller target dataset. This is particularly useful when the target dataset cannot be used to train a model from scratch due to lack of data. This second algorithm, which we named Cascade Transfer Learning (CTC), yields similar memory advantages to Deep Cascade Learning, and enables minimal computational complexity for feature transfer. In addition, CTC provides competitive results in comparison with other transfer learning approaches.

Finally, we further explore the scalability of Deep Cascade Learning by executing it on a multi-variate time series classification task. Such tasks include predicting human activities from body-worn sensors. Deep Cascade Learning can be used to reduce the training time of these models, opening up the possibility of online training on smart devices.
University of Southampton
Marquez, Enrique S.
e0489457-4bcb-406a-845a-2f33d2221e93
Marquez, Enrique S.
e0489457-4bcb-406a-845a-2f33d2221e93
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9

Marquez, Enrique S. (2019) Deep cascade learning. University of Southampton, Doctoral Thesis, 120pp.

Record type: Thesis (Doctoral)

Abstract

Deep Learning has demonstrated outstanding performance on several machine learning tasks. These results are attributed to training very deep networks on large scale datasets. In this thesis we investigate training models in a layer-wise fashion. We quantify performance and discuss the advantages of using such training algorithms on computer vision and signal processing tasks.

Inspired by the Cascade Correlation algorithm, which is a growing neural network that iteratively learns artificial neurons, we developed a supervised layer-wise training algorithm, which we name Deep Cascade Learning. Our methodology takes as input the architecture to train and splits the model in submodels, where each iteration trains only one layer of the network. The feature representation gets more robust as layers are stacked. Moreover, the algorithm provides training complexity reduction while preserving competitive results in comparison with state-of-the-art end to end training. We demonstrate these advantages on multiple benchmark datasets.

Given that Deep Cascade Learning trains models from scratch successfully, we also look at layer-wise methods to transfer features from a large base dataset, to a smaller target dataset. This is particularly useful when the target dataset cannot be used to train a model from scratch due to lack of data. This second algorithm, which we named Cascade Transfer Learning (CTC), yields similar memory advantages to Deep Cascade Learning, and enables minimal computational complexity for feature transfer. In addition, CTC provides competitive results in comparison with other transfer learning approaches.

Finally, we further explore the scalability of Deep Cascade Learning by executing it on a multi-variate time series classification task. Such tasks include predicting human activities from body-worn sensors. Deep Cascade Learning can be used to reduce the training time of these models, opening up the possibility of online training on smart devices.

Text
Thesis_EM - Version of Record
Available under License University of Southampton Thesis Licence.
Download (6MB)

More information

Published date: June 2019

Identifiers

Local EPrints ID: 433532
URI: http://eprints.soton.ac.uk/id/eprint/433532
PURE UUID: 8baa03de-0773-4bc0-983a-9bfeb074b315
ORCID for Jonathon Hare: ORCID iD orcid.org/0000-0003-2921-4283

Catalogue record

Date deposited: 27 Aug 2019 16:30
Last modified: 16 Mar 2024 03:50

Export record

Contributors

Author: Enrique S. Marquez
Thesis advisor: Jonathon Hare ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×