The University of Southampton
University of Southampton Institutional Repository

Pruning resnet neural networks block by block

Pruning resnet neural networks block by block
Pruning resnet neural networks block by block
Neural network pruning has gained popularity for deep models with the goal of reducing storage and computational requirements, both for inference and training. Pruning weights from neural networks to make models sparse has been discussed for a long time starting with in 1990s. Structured pruning is more recent and it aims to minimize the size of neural networks as opposed to only introducing sparsity, having the advantage of not requiring specialized hardware or software. Most structured pruning works focus on neurons or convolutional filters. ResNet is a model architecture composed of many blocks that are linked with residual connections. In our work, we explore pruning larger structures – ResNet blocks, and thoroughly study the feasibility of block by block pruning whilst keeping track of the cost of fine-tuning the pruned networks. We use different block saliency metrics as well as different fine-tuning schedules and parameters. Pruning 27 blocks (50%) from a ResNet-110, in our best configuration, gives 6.48% test error on CIFAR-10, a 0.45% loss from the initial model. When pruning 45 blocks to obtain a similar size to that of a ResNet-20, our best method has a 1.92% loss from initial, 7.95% error. We observe that training small, standard ResNet configurations gives better results than pruning and argue that pruning block by block is only effective for pruning a small number of blocks, or when starting with a model pre-trained elsewhere and the cost of fine-tuning is of concern (fine-tuning alone can be much cheaper than training from scratch and give acceptable results, depending on the pruning configuration). Finally, our pruning work is produced from hundreds of experiments, and a by-product of running, organising and analysing them is our experiment management framework, dbx, aims to simplify this process by storing experiments and their results (as logs) in repositories that can be queried and synchronized between computers.
University of Southampton
Velici, Vlad, Sebastian
9c9e1a57-8667-4239-a31d-234da5ce9f4b
Velici, Vlad, Sebastian
9c9e1a57-8667-4239-a31d-234da5ce9f4b
Prugel-Bennett, Adam
b107a151-1751-4d8b-b8db-2c395ac4e14e

Velici, Vlad, Sebastian (2022) Pruning resnet neural networks block by block. University of Southampton, Doctoral Thesis, 135pp.

Record type: Thesis (Doctoral)

Abstract

Neural network pruning has gained popularity for deep models with the goal of reducing storage and computational requirements, both for inference and training. Pruning weights from neural networks to make models sparse has been discussed for a long time starting with in 1990s. Structured pruning is more recent and it aims to minimize the size of neural networks as opposed to only introducing sparsity, having the advantage of not requiring specialized hardware or software. Most structured pruning works focus on neurons or convolutional filters. ResNet is a model architecture composed of many blocks that are linked with residual connections. In our work, we explore pruning larger structures – ResNet blocks, and thoroughly study the feasibility of block by block pruning whilst keeping track of the cost of fine-tuning the pruned networks. We use different block saliency metrics as well as different fine-tuning schedules and parameters. Pruning 27 blocks (50%) from a ResNet-110, in our best configuration, gives 6.48% test error on CIFAR-10, a 0.45% loss from the initial model. When pruning 45 blocks to obtain a similar size to that of a ResNet-20, our best method has a 1.92% loss from initial, 7.95% error. We observe that training small, standard ResNet configurations gives better results than pruning and argue that pruning block by block is only effective for pruning a small number of blocks, or when starting with a model pre-trained elsewhere and the cost of fine-tuning is of concern (fine-tuning alone can be much cheaper than training from scratch and give acceptable results, depending on the pruning configuration). Finally, our pruning work is produced from hundreds of experiments, and a by-product of running, organising and analysing them is our experiment management framework, dbx, aims to simplify this process by storing experiments and their results (as logs) in repositories that can be queried and synchronized between computers.

Text
Vlad-Velici-Thesis - Version of Record
Available under License University of Southampton Thesis Licence.
Download (3MB)
Text
Permission to deposit thesis - form
Restricted to Repository staff only
Available under License University of Southampton Thesis Licence.

More information

Published date: 23 January 2022

Identifiers

Local EPrints ID: 467305
URI: http://eprints.soton.ac.uk/id/eprint/467305
PURE UUID: 755cd1c5-541b-433b-ad86-3cd9ddc573d7
ORCID for Vlad, Sebastian Velici: ORCID iD orcid.org/0000-0002-1549-0116

Catalogue record

Date deposited: 05 Jul 2022 17:00
Last modified: 16 Mar 2024 17:32

Export record

Contributors

Author: Vlad, Sebastian Velici ORCID iD
Thesis advisor: Adam Prugel-Bennett

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×