The University of Southampton
University of Southampton Institutional Repository

Dynamic DNNs meet runtime resource management on mobile and embedded platforms

Dynamic DNNs meet runtime resource management on mobile and embedded platforms
Dynamic DNNs meet runtime resource management on mobile and embedded platforms
Deep neural network (DNN) inference is increasingly being executed on mobile and embedded platforms due to low latency and better privacy. However, efficient deployment on these platforms is challenging due to the intensive computation and memory access. We propose a holistic system design for DNN performance and energy optimisation, combining the trade-off opportunities in both algorithms and hardware. The system can be viewed as three abstract layers: the device layer contains heterogeneous computing resources; the application layer has multiple concurrent workloads; and the runtime resource management layer monitors the dynamically changing algorithms' performance targets as well as hardware resources and constraints, and tries to meet them by tuning the algorithm and hardware at the same time. Moreover, We illustrate the runtime approach through a dynamic version of 'once-for-all network' (namely Dynamic-OFA), which can scale the ConvNet architecture to fit heterogeneous computing resources efficiently and has good generalisation for different model architectures such as Transformer. Compared to the state-of-the-art Dynamic DNNs, our experimental results using ImageNet on a Jetson Xavier NX show that the Dynamic-OFA is up to 3.5x (CPU), 2.4x (GPU) faster for similar ImageNet Top-1 accuracy, or 3.8% (CPU), 5.1% (GPU) higher accuracy at similar latency. Furthermore, compared with Linux governor (e.g. performance, schedutil), our runtime approach reduces the energy consumption by 16.5% at similar latency.
Xun, Lei
51a0da82-6979-49a8-8eff-ada011f5aff5
Al-Hashimi, Bashir
32946be3-e73a-4742-b5cd-8e840748aad2
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Merrett, Geoff
89b3a696-41de-44c3-89aa-b0aa29f54020
Xun, Lei
51a0da82-6979-49a8-8eff-ada011f5aff5
Al-Hashimi, Bashir
32946be3-e73a-4742-b5cd-8e840748aad2
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Merrett, Geoff
89b3a696-41de-44c3-89aa-b0aa29f54020

Xun, Lei, Al-Hashimi, Bashir, Hare, Jonathon and Merrett, Geoff (2022) Dynamic DNNs meet runtime resource management on mobile and embedded platforms. UK Mobile, Wearable and Ubiquitous Systems Research Symposium 2022, UCL, London, United Kingdom. 04 - 05 Jul 2022. (In Press)

Record type: Conference or Workshop Item (Other)

Abstract

Deep neural network (DNN) inference is increasingly being executed on mobile and embedded platforms due to low latency and better privacy. However, efficient deployment on these platforms is challenging due to the intensive computation and memory access. We propose a holistic system design for DNN performance and energy optimisation, combining the trade-off opportunities in both algorithms and hardware. The system can be viewed as three abstract layers: the device layer contains heterogeneous computing resources; the application layer has multiple concurrent workloads; and the runtime resource management layer monitors the dynamically changing algorithms' performance targets as well as hardware resources and constraints, and tries to meet them by tuning the algorithm and hardware at the same time. Moreover, We illustrate the runtime approach through a dynamic version of 'once-for-all network' (namely Dynamic-OFA), which can scale the ConvNet architecture to fit heterogeneous computing resources efficiently and has good generalisation for different model architectures such as Transformer. Compared to the state-of-the-art Dynamic DNNs, our experimental results using ImageNet on a Jetson Xavier NX show that the Dynamic-OFA is up to 3.5x (CPU), 2.4x (GPU) faster for similar ImageNet Top-1 accuracy, or 3.8% (CPU), 5.1% (GPU) higher accuracy at similar latency. Furthermore, compared with Linux governor (e.g. performance, schedutil), our runtime approach reduces the energy consumption by 16.5% at similar latency.

Text
Dynamic DNNs Meet Runtime Resource Management on Mobile and Embedded Platforms - Accepted Manuscript
Available under License Creative Commons Attribution.
Download (119kB)

More information

Accepted/In Press date: 17 May 2022
Venue - Dates: UK Mobile, Wearable and Ubiquitous Systems Research Symposium 2022, UCL, London, United Kingdom, 2022-07-04 - 2022-07-05

Identifiers

Local EPrints ID: 457582
URI: http://eprints.soton.ac.uk/id/eprint/457582
PURE UUID: dab0631f-5803-4770-a1ab-223f6e89439b
ORCID for Jonathon Hare: ORCID iD orcid.org/0000-0003-2921-4283
ORCID for Geoff Merrett: ORCID iD orcid.org/0000-0003-4980-3894

Catalogue record

Date deposited: 13 Jun 2022 16:45
Last modified: 23 Jul 2022 01:56

Export record

Contributors

Author: Lei Xun
Author: Bashir Al-Hashimi
Author: Jonathon Hare ORCID iD
Author: Geoff Merrett ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×