The University of Southampton
University of Southampton Institutional Repository

Using the provenance from astronomical workflows to increase processing efficiency

Using the provenance from astronomical workflows to increase processing efficiency
Using the provenance from astronomical workflows to increase processing efficiency

Astronomy is increasingly becoming a data-driven science as the community builds larger instruments which are capable of gathering more data than previously possible. As the sizes of the datasets increase, it becomes even more important to make the most efficient use of the computational resources available. In this work, we highlight how provenance can be used to increase the computational efficiency of astronomical workflows. We describe a provenance-enabled image processing pipeline and motivate the generation of provenance with two relevant use cases. The first use case investigates the origin of an optical variation and the second is concerned with the objects used to calibrate the image. The provenance was then queried in order to evaluate the relative computational efficiency of use case evaluation, with and without the use of provenance. We find that recording the provenance of the pipeline increases the original processing time by ~45%. However, we find that when evaluating the two identified use cases, the inclusion of provenance improves the efficiency of processing by ~99% and ~96% for Use Cases 1 and 2, respectively. Furthermore, we combine these results with the probability that Use Cases 1 and 2 will need to be evaluated and find a net decrease in computational processing efficiency of 13–44% when incorporating provenance generation within the workflow. However, we deduce that provenance has the potential to produce a net increase in this efficiency if more uses cases are to be considered.

0302-9743
101-112
Springer
Johnson, Michael A.C.
33a0d8cb-491b-4b3f-b193-540a331ac705
Moreau, Luc
033c63dd-3fe9-4040-849f-dfccbe0406f8
Chapman, Adriane
721b7321-8904-4be2-9b01-876c430743f1
Gandhi, Poshak
5bc3b5af-42b0-4dd8-8f1f-f74048d4d4a9
Sáenz-Adán, Carlos
c752e854-e029-4190-964b-219053afd11a
Johnson, Michael A.C.
33a0d8cb-491b-4b3f-b193-540a331ac705
Moreau, Luc
033c63dd-3fe9-4040-849f-dfccbe0406f8
Chapman, Adriane
721b7321-8904-4be2-9b01-876c430743f1
Gandhi, Poshak
5bc3b5af-42b0-4dd8-8f1f-f74048d4d4a9
Sáenz-Adán, Carlos
c752e854-e029-4190-964b-219053afd11a

Johnson, Michael A.C., Moreau, Luc, Chapman, Adriane, Gandhi, Poshak and Sáenz-Adán, Carlos (2018) Using the provenance from astronomical workflows to increase processing efficiency. In Provenance and Annotation of Data and Processes - 7th International Provenance and Annotation Workshop, IPAW 2018, Proceedings. vol. 11017 LNCS, Springer. pp. 101-112 . (doi:10.1007/978-3-319-98379-0_8).

Record type: Conference or Workshop Item (Paper)

Abstract

Astronomy is increasingly becoming a data-driven science as the community builds larger instruments which are capable of gathering more data than previously possible. As the sizes of the datasets increase, it becomes even more important to make the most efficient use of the computational resources available. In this work, we highlight how provenance can be used to increase the computational efficiency of astronomical workflows. We describe a provenance-enabled image processing pipeline and motivate the generation of provenance with two relevant use cases. The first use case investigates the origin of an optical variation and the second is concerned with the objects used to calibrate the image. The provenance was then queried in order to evaluate the relative computational efficiency of use case evaluation, with and without the use of provenance. We find that recording the provenance of the pipeline increases the original processing time by ~45%. However, we find that when evaluating the two identified use cases, the inclusion of provenance improves the efficiency of processing by ~99% and ~96% for Use Cases 1 and 2, respectively. Furthermore, we combine these results with the probability that Use Cases 1 and 2 will need to be evaluated and find a net decrease in computational processing efficiency of 13–44% when incorporating provenance generation within the workflow. However, we deduce that provenance has the potential to produce a net increase in this efficiency if more uses cases are to be considered.

Text
Provenance_Astronomical_Workflows - Accepted Manuscript
Download (491kB)

More information

e-pub ahead of print date: 6 September 2018
Venue - Dates: 7th International Provenance and Annotation Workshop, IPAW 2018, , London, United Kingdom, 2018-07-09 - 2018-07-10

Identifiers

Local EPrints ID: 425165
URI: http://eprints.soton.ac.uk/id/eprint/425165
ISSN: 0302-9743
PURE UUID: 304afad8-fc43-42a8-bf5a-61d50f260486
ORCID for Michael A.C. Johnson: ORCID iD orcid.org/0000-0002-5566-6147
ORCID for Luc Moreau: ORCID iD orcid.org/0000-0002-3494-120X
ORCID for Adriane Chapman: ORCID iD orcid.org/0000-0002-3814-2587
ORCID for Poshak Gandhi: ORCID iD orcid.org/0000-0003-3105-2615

Catalogue record

Date deposited: 11 Oct 2018 16:30
Last modified: 06 Jun 2024 04:04

Export record

Altmetrics

Contributors

Author: Michael A.C. Johnson ORCID iD
Author: Luc Moreau ORCID iD
Author: Adriane Chapman ORCID iD
Author: Poshak Gandhi ORCID iD
Author: Carlos Sáenz-Adán

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×