The University of Southampton
University of Southampton Institutional Repository

Elastic symbiotic scaling of operators and resources in stream processing systems

Elastic symbiotic scaling of operators and resources in stream processing systems
Elastic symbiotic scaling of operators and resources in stream processing systems
Distributed stream processing frameworks are designed to perform continuous computation on possibly unbounded data streams whose rates can change over time. Devising solutions to make such systems elastically scale is a fundamental goal to achieve desired performance and cut costs caused by resource over-provisioning. These systems can be scaled along two dimensions: the operator parallelism and the number of resources. In this paper, we show how these two dimensions, as two symbiotic entities, are independent but must mutually interact for the global benefit of the system. On the basis of this observation, we propose a fine-grained model for estimating the resource utilization of a stream processing application that enables the independent scaling of operators and resources. A simple, yet effective, combined management of the two dimensions allows us to propose ELYSIUM, a novel elastic scaling approach that provides efficient resource utilization. We implemented the proposed approach within Apache Storm and tested it by running two real-world applications with different input load curves. The outcomes backup our claims showing that the proposed symbiotic management outperforms elastic scaling strategies where operators and resources are jointly scaled.
1045-9219
572-585
Lombardi, Federico
78e41297-64c9-4c1e-9515-8eb59334a795
Aniello, Leonardo
9846e2e4-1303-4b8b-9092-5d8e9bb514c3
Bonomi, Silvia
776b25cf-fe55-4e7e-80cc-32ea9f7e010c
Querzoni, Leonardo
c0eee656-74e7-419d-876c-3cad808683d6
Lombardi, Federico
78e41297-64c9-4c1e-9515-8eb59334a795
Aniello, Leonardo
9846e2e4-1303-4b8b-9092-5d8e9bb514c3
Bonomi, Silvia
776b25cf-fe55-4e7e-80cc-32ea9f7e010c
Querzoni, Leonardo
c0eee656-74e7-419d-876c-3cad808683d6

Lombardi, Federico, Aniello, Leonardo, Bonomi, Silvia and Querzoni, Leonardo (2018) Elastic symbiotic scaling of operators and resources in stream processing systems. IEEE Transactions on Parallel and Distributed Systems, 29 (3), 572-585. (doi:10.1109/TPDS.2017.2762683).

Record type: Article

Abstract

Distributed stream processing frameworks are designed to perform continuous computation on possibly unbounded data streams whose rates can change over time. Devising solutions to make such systems elastically scale is a fundamental goal to achieve desired performance and cut costs caused by resource over-provisioning. These systems can be scaled along two dimensions: the operator parallelism and the number of resources. In this paper, we show how these two dimensions, as two symbiotic entities, are independent but must mutually interact for the global benefit of the system. On the basis of this observation, we propose a fine-grained model for estimating the resource utilization of a stream processing application that enables the independent scaling of operators and resources. A simple, yet effective, combined management of the two dimensions allows us to propose ELYSIUM, a novel elastic scaling approach that provides efficient resource utilization. We implemented the proposed approach within Apache Storm and tested it by running two real-world applications with different input load curves. The outcomes backup our claims showing that the proposed symbiotic management outperforms elastic scaling strategies where operators and resources are jointly scaled.

Text
elysium_main - Version of Record
Download (7MB)

More information

Accepted/In Press date: 29 September 2017
e-pub ahead of print date: 13 October 2017
Published date: 1 March 2018

Identifiers

Local EPrints ID: 423352
URI: https://eprints.soton.ac.uk/id/eprint/423352
ISSN: 1045-9219
PURE UUID: e1a24bd9-717b-42f6-9f40-decd97c5447b
ORCID for Federico Lombardi: ORCID iD orcid.org/0000-0001-6463-8722

Catalogue record

Date deposited: 20 Sep 2018 16:30
Last modified: 12 Nov 2019 01:25

Export record

Altmetrics

Contributors

Author: Federico Lombardi ORCID iD
Author: Leonardo Aniello
Author: Silvia Bonomi
Author: Leonardo Querzoni

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×