Deep cascade learning

Abstract

Deep Learning has demonstrated outstanding performance on several machine learning tasks. These results are attributed to training very deep networks on large scale datasets. In this thesis we investigate training models in a layer-wise fashion. We quantify performance and discuss the advantages of using such training algorithms on computer vision and signal processing tasks.

Inspired by the Cascade Correlation algorithm, which is a growing neural network that iteratively learns artificial neurons, we developed a supervised layer-wise training algorithm, which we name Deep Cascade Learning. Our methodology takes as input the architecture to train and splits the model in submodels, where each iteration trains only one layer of the network. The feature representation gets more robust as layers are stacked. Moreover, the algorithm provides training complexity reduction while preserving competitive results in comparison with state-of-the-art end to end training. We demonstrate these advantages on multiple benchmark datasets.

Given that Deep Cascade Learning trains models from scratch successfully, we also look at layer-wise methods to transfer features from a large base dataset, to a smaller target dataset. This is particularly useful when the target dataset cannot be used to train a model from scratch due to lack of data. This second algorithm, which we named Cascade Transfer Learning (CTC), yields similar memory advantages to Deep Cascade Learning, and enables minimal computational complexity for feature transfer. In addition, CTC provides competitive results in comparison with other transfer learning approaches.

Finally, we further explore the scalability of Deep Cascade Learning by executing it on a multi-variate time series classification task. Such tasks include predicting human activities from body-worn sensors. Deep Cascade Learning can be used to reduce the training time of these models, opening up the possibility of online training on smart devices.

More information

Identifiers

ORCID for Jonathon Hare: orcid.org/0000-0003-2921-4283

Catalogue record

Export record